Nagios deep dive: retention.dat and modified_attributes · 23 November 2010, 07:26
When Nagios core (the daemon, typically started by a script in /etc/init.d/) starts up, it follows a rather involved process to turn the configuration files and domain-specific language (DSL) contained within them into in-memory objects – the 10000 foot view of this process is:
- Parse and validate nagios.cfg
- Parse all text-based configuration files (or read the specified objects pre-cache file) based on the cfg_file, cfg_fir, precached_object_file directive in nagios.cfg and command line options passed to Nagios, validate syntax of all files as the files are read.
- Parse retention.dat and load desired persistent object attributes into memory for any objects that exist based on the flat configuration file or objects.pre-cache contents read in from previous steps (any objects stored in retention.dat that do not have counterparts in the objects pre-cache file or Nagios DSL-based configuration files are ignored).
modified_attributes tells Nagios which attributes of an object should be loaded into memory as Nagios reads object state from retention.dat; the code that uses this field (all DSL-related code is in the xdata/ directory of the source tree) uses bit-shifting to store and determine which attributes should be read into memory for an object and which should be ignored.
#define MODATTR_NONE 0 #define MODATTR_NOTIFICATIONS_ENABLED 1 #define MODATTR_ACTIVE_CHECKS_ENABLED 2 #define MODATTR_PASSIVE_CHECKS_ENABLED 4 #define MODATTR_EVENT_HANDLER_ENABLED 8 #define MODATTR_FLAP_DETECTION_ENABLED 16 #define MODATTR_FAILURE_PREDICTION_ENABLED 32 #define MODATTR_PERFORMANCE_DATA_ENABLED 64 #define MODATTR_OBSESSIVE_HANDLER_ENABLED 128 #define MODATTR_EVENT_HANDLER_COMMAND 256 #define MODATTR_CHECK_COMMAND 512 #define MODATTR_NORMAL_CHECK_INTERVAL 1024 #define MODATTR_RETRY_CHECK_INTERVAL 2048 #define MODATTR_MAX_CHECK_ATTEMPTS 4096 #define MODATTR_FRESHNESS_CHECKS_ENABLED 8192 #define MODATTR_CHECK_TIMEPERIOD 16384 #define MODATTR_CUSTOM_VARIABLE 32768 #define MODATTR_NOTIFICATION_TIMEPERIOD 65536
The default value for modified_attributes is 0 – ignore all attributes from retention.dat that have counterpart constants in common.h
When an object’s state for the fields listed is changed as Nagios runs, Nagios changes the value of the modified_attributes field to include the constant that represents the field; this allows the retention.dat parsing code to know which attributes to read into memory as an object is parsed from retention.dat into memory when Nagios starts.
A common use case showing this process:
- User logs into the Nagios UI and disables active checks for a host along with notifications
When these two actions are processed, Nagios core will then change modified_attributes to indicate that the state of the notifications_enabled and active_checks_enabled fields were changed from their default values by setting modified_attributes to 3, which is the result of code similar to this:
modified_attributes |= MODATTR_NOTIFICATIONS_ENABLED modified_attributes |= MODATTR_ACTIVE_CHECKS_ENABLED
When Nagios is stopped, it serializes all objects from memory to disk – the modified_attributes attribute is one of the attributes written to disk.
Our team has taken the approach of writing out our own retention.dat files based on state for Nagios objects stored in a database as a part of our current distributed nagios implementation – knowing how modified_attributes works fixed a long standing bug in our code that was causing attributes for hosts and services that had been modified in-flight to be ignored when Nagios started – we hope this short article helps you avoid the same bug.
Special thanks to my managers Mike Fischer and Eric Scholz at Comcast (a great place to work as a developer!) for allowing me to share information learned while at work based on our use of open source software with the community – and special thanks to Ryan Richins for his work with me on uncovering the cause of this bug in our custom Nagios configuration distribution code.
— Max Schubert