Icinga2 graphing with InfluxDB and Grafana

Written by Claudio Kuenzler - 1 comments

Published on December 8th 2017 - last updated on April 25th 2024 - Listed in Icinga Monitoring Influx Database

---

Note: This article covers InfluxDB v1 with IQL (Influx Query Language). The newer InfluxDB 2 uses a different query language (Flux) which is not covered here.

As you can imagine from my previous post (icinga2-classicui shows Whoops error on new Icinga 2.8 install) I'm currently setting up a renewed monitoring architecture running on Ubuntu 16.04 Xenial.

To my big surprise PNP4Nagios doesn't run anymore on Xenial because Xenial ships with PHP 7.0 but PNP4Nagios is now running with PHP 7.0 (see this issue on github). There are surely ways around it like installing special PHP repositories to install older PHP versions, but I decided to test another graphing method: Grafana as user interface (grapher) and InfluxDB as database. 

Note in advance: Getting your graphs shown in Grafana certainly requires much more work than with PNP4Nagios, especially if you heavily rely on nrpe checks like me.

Part I: Preparation of InfluxDB

Let's install the InfluxDB right from the Ubuntu repos:

root@inf-mon02-t:~# apt-get install influxdb influxdb-client

After the installation, influxdb's daemon (influxd) is launched automatically and is listening:

root@inf-mon02-t:~# netstat -lntup | grep influx
tcp        0      0 127.0.0.1:8091          0.0.0.0:*               LISTEN      25027/influxd
tcp6       0      0 :::8083                 :::*                    LISTEN      25027/influxd
tcp6       0      0 :::8086                 :::*                    LISTEN      25027/influxd
tcp6       0      0 :::8088                 :::*                    LISTEN      25027/influxd

Data will be stored in /var/lib/influxdb:

root@inf-mon02-t:~# ll /var/lib/influxdb/
total 16
drwxr-xr-x 3 influxdb influxdb 4096 Dec  8 12:11 data
drwx------ 2 influxdb influxdb 4096 Dec  8 12:11 hh
drwxr-xr-x 3 influxdb influxdb 4096 Dec  8 12:11 meta
drwx------ 3 influxdb influxdb 4096 Dec  8 12:11 wal

InfluxDB can now be accessed using the InfluxDB client (influx):

root@inf-mon02-t:~# influx
Visit https://enterprise.influxdata.com to register for updates, InfluxDB server management, and monitoring.
Connected to http://localhost:8086 version 0.10.0
InfluxDB shell 0.10.0
> SHOW DATABASES
name: databases
---------------
name
_internal

> quit

Let's prepare a database for Icinga2 and create a database user for it:

root@inf-mon02-t:~# influx
Visit https://enterprise.influxdata.com to register for updates, InfluxDB server management, and monitoring.
Connected to http://localhost:8086 version 0.10.0
InfluxDB shell 0.10.0
> CREATE DATABASE icinga2
> SHOW DATABASES
name: databases
---------------
name
_internal
icinga2

> CREATE USER icinga2 WITH PASSWORD 'mysupersecretpassword'
> GRANT ALL ON icinga2 TO icinga2
> SHOW GRANTS FOR icinga2
database        privilege
icinga2         ALL PRIVILEGES

> quit

Part II: Tell Icinga2 to use InfluxDB for performance data

The following Icinga2 features need to be enabled:

root@inf-mon02-t:~# icinga2 feature enable influxdb
root@inf-mon02-t:~# icinga2 feature enable perfdata

This requires a restart of Icinga2:

root@inf-mon02-t:~# service icinga2 restart

Verify, that the features are really enabled:

root@inf-mon02-t:~# icinga2 feature list
Disabled features: debuglog elasticsearch gelf graphite ido-mysql livestatus opentsdb syslog
Enabled features: api checker command compatlog influxdb mainlog notification perfdata statusdata

In theory that should already be enough. From the official Icinga2 documentation:

"By default the InfluxdbWriter feature expects the InfluxDB daemon to listen at 127.0.0.1 on port 8086."

Important to add a note here: The default database Icinga2 wants to write into is called "icinga2".

Now let's check if InfluxDB is really receiving data from Icinga2's.

Note: InfluxDB doesn't use the term "table" as classical SQL databases. This is called a "measurement" in InfluxDB terminology and is "comparable" to a table. It's my first day with InfluxDB so this is very likely not technically correct, but helpful to understand the query syntax.

root@inf-mon02-t:~# influx
Visit https://enterprise.influxdata.com to register for updates, InfluxDB server management, and monitoring.
Connected to http://localhost:8086 version 0.10.0
InfluxDB shell 0.10.0
> USE icinga2
Using database icinga2
> SHOW MEASUREMENTS
name: measurements
------------------
name
apt
disk
hostalive
http
icinga
load
ping4
ping6
procs
ssh
swap
users

> SELECT * FROM ping4
name: ping4
-----------
time                    hostname        metric  service value
1512732076000000000     inf-jira01-t    pl      ping4   0
1512732076000000000     inf-jira01-t    rta     ping4   0.003722
1512732076000000000     inf-mon02-t     pl      ping4   0
1512732076000000000     inf-mon02-t     rta     ping4   4.1e-05
1512732138000000000     inf-mon02-t     rta     ping4   4.6e-05
1512732138000000000     inf-jira01-t    rta     ping4   0.00366
1512732138000000000     inf-mon02-t     pl      ping4   0
1512732138000000000     inf-jira01-t    pl      ping4   0
1512732200000000000     inf-mon02-t     pl      ping4   0
1512732200000000000     inf-mon02-t     rta     ping4   5.5e-05
1512732200000000000     inf-jira01-t    rta     ping4   0.003685
1512732200000000000     inf-jira01-t    pl      ping4   0
1512732262000000000     inf-jira01-t    rta     ping4   0.003784
1512732262000000000     inf-mon02-t     pl      ping4   0

> quit

Yes, there's data in there!

But all this collected data is only helpful if we can create graphs from it. Grafana is able to connect to an InfluxDB and create graphs from the data, so let's do this!

Part III: The fight to connect Grafana to InfluxDB

The installation of Grafana is very simple as Xenial already has Grafana packages in their repos.

root@inf-mon02-t:~# apt-get install grafana grafana-data

Grafana is listening on port tcp/3000:

root@inf-mon02-t:~# netstat -lntup|grep grafana
tcp6       0      0 :::3000                 :::*                    LISTEN      27579/grafana

This is the listener port of the Grafana-internal webserver. This is how Grafana looks in the browser:

The default user and password is admin/admin and the default (and yet empty) dashboard is shown:

To add the InfluxDB as a data source, click on Data Sources -> Add new:

Set a name for this new data source. I called it "InfluxDB-Icinga2".
On Ubuntu 16.04 Xenial an InfluxDB 0.9.x is installed, so set "Type" to "InfluxDB 0.9.x". 
In "Http settings" set the access to the InfluxDB's API port, which is "http://localhost:8086" and has a "direct" access. 
Grafana requires given user and password to connect to InfluxDB. Use the icinga2 user which was created above.

Once the data source was addded, I clicked on "Test Connection" and got the following error in Chrome:

Unknown error
InfluxDB Error: Cannot read property 'message' of null

In Firefox I got a slightly different error:

Unknown error
InfluxDB Error: err.data is null

I wasn't able to find a solution to this, but it led me to believe that a lot of bugs have been fixed since the Grafana version for Xenial (2.6.0) was released.
The current version is 4.6.2 so I decided to install this one. But first, Grafana (from the Ubuntu repos) needs to be uninstalled again:

root@inf-mon02-t:~# apt-get remove grafana grafana-data
root@inf-mon02-t:~# apt-get purge grafana grafana-data
root@inf-mon02-t:~# rm -rf /usr/share/grafana

I downloaded the current release's Debian package and installed it:

root@inf-mon02-t:~# wget https://s3-us-west-2.amazonaws.com/grafana-releases/release/grafana_4.6.2_amd64.deb
root@inf-mon02-t:~# sudo dpkg -i grafana_4.6.2_amd64.deb

At the end of the installation of the deb package, the following notes show up:

### NOT starting on installation, please execute the following statements to configure grafana to start automatically using systemd
 sudo /bin/systemctl daemon-reload
 sudo /bin/systemctl enable grafana-server
### You can start grafana-server by executing
 sudo /bin/systemctl start grafana-server

Hmm... could have been placed into the deb package, too. 
So let's do this:

root@inf-mon02-t:~# sudo /bin/systemctl daemon-reload
root@inf-mon02-t:~# sudo /bin/systemctl enable grafana-server
Synchronizing state of grafana-server.service with SysV init with /lib/systemd/systemd-sysv-install...
Executing /lib/systemd/systemd-sysv-install enable grafana-server
Created symlink from /etc/systemd/system/multi-user.target.wants/grafana-server.service to /usr/lib/systemd/system/grafana-server.service.
root@inf-mon02-t:~# sudo /bin/systemctl start grafana-server

And yes, Grafana is started:

root@inf-mon02-t:~# netstat -lntup| grep grafana
tcp6       0      0 :::3000                 :::*                    LISTEN      29922/grafana-serve

Back to the login with the browser. Wow - the interface has really changed a lot:

Add a comment

Show form to leave a comment

Comments (newest first)

Cedric from Paris wrote on Feb 7th, 2024:

Merci Beaucoup pour votre tutoriel, ça ma beaucoup aider, j'ai suivi la procédure et ça bien marche.
Concernant grafana, je me suis basé sur la nouvelle version

---