On a new Icinga 2 monitoring server with Nagiosgraph, I became aware of a huge nagiosgraph.log file. A short look into the log file revealed hundreds and hundreds of such errors:
Mon Nov 23 12:29:02 2015 insert.pl 21629 error RRDs::update ERR /var/nagiosgraph/rrd/myhost/HTTP___size.rrd_min: illegal attempt to update using time 1448278091 when last update time is 1448278131 (minimum one second step)
Mon Nov 23 12:29:02 2015 insert.pl 21629 error ds = [
'/var/nagiosgraph/rrd/myhost/HTTP___size.rrd_min',
'1448278091:838:0'
];
The error message itself explains the problem pretty well: The RRD file cannot be updated because the data with the current timestamp (1448278091) is older than the existing last entry (1448278131). But how can this happen?
Actually I built this problem on my own. When you take a look at my article How to integrate Nagiosgraph in Icinga 2, you see that I created a wrapper-script (run as a cronjob) which searches for the perfdata logs and combines them. As my search command I used the following find command:
find $perfdatadir/ -type f -regex ".*[0-9]"
/var/spool/icinga2/nagiosgraph/nagiosgraph-host-perfdata.1448278031
/var/spool/icinga2/nagiosgraph/nagiosgraph-host-perfdata.1448278046
/var/spool/icinga2/nagiosgraph/nagiosgraph-perfdata.1448278046
/var/spool/icinga2/nagiosgraph/nagiosgraph-perfdata.1448278031
The script then goes through all the perfdata log files (in the order found by the find command) and combines them. As you can see above, the perfdata file with timestamp 1448278046 was found BEFORE the file with timestamp 1448278031. So the content of the newer file is parsed BEFORE the content of the older file. Hence the error message from Nagiosgraph.
I modified my find command to sort the found files by age:
find $perfdatadir/ -type f -regex ".*[0-9]" -exec ls -tr {} +
/var/spool/icinga2/nagiosgraph/nagiosgraph-perfdata.1448278811
/var/spool/icinga2/nagiosgraph/nagiosgraph-host-perfdata.1448278811
/var/spool/icinga2/nagiosgraph/nagiosgraph-perfdata.1448278826
/var/spool/icinga2/nagiosgraph/nagiosgraph-host-perfdata.1448278826
/var/spool/icinga2/nagiosgraph/nagiosgraph-perfdata.1448278841
/var/spool/icinga2/nagiosgraph/nagiosgraph-host-perfdata.1448278841
And now the errors are gone.
No comments yet.
AWS Android Ansible Apache Apple Atlassian BSD Backup Bash Bluecoat CMS Chef Cloud Coding Consul Containers CouchDB DB DNS Database Databases Docker ELK Elasticsearch Filebeat FreeBSD Galera Git GlusterFS Grafana Graphics HAProxy HTML Hacks Hardware Icinga Influx Internet Java KVM Kibana Kodi Kubernetes LVM LXC Linux Logstash Mac Macintosh Mail MariaDB Minio MongoDB Monitoring Multimedia MySQL NFS Nagios Network Nginx OSSEC OTRS Office PGSQL PHP Perl Personal PostgreSQL Postgres PowerDNS Proxmox Proxy Python Rancher Rant Redis Roundcube SSL Samba Seafile Security Shell SmartOS Solaris Surveillance Systemd TLS Tomcat Ubuntu Unix VMWare VMware Varnish Virtualization Windows Wireless Wordpress Wyse ZFS Zoneminder