SSL TLS SNI certificate monitoring with Icingaweb2 x509 module

Written by Claudio Kuenzler - 2 comments

Published on January 7th 2019 - last updated on November 20th 2020 - Listed in Monitoring SSL TLS Icinga

---

Back in November 2018 I attended the Open Source Monitoring Conference (OSMC) in Nuremberg, Germany. One of the presentations included a quick demo of a new module for Icingaweb2, the "quasi standard" user interface for Icinga2. The module was called "x509" and it's purpose, as one may already think when reading the name, is to monitor SSL/TLS certificates. 

This article will cover the installation, activation, configuration of the module. And some first impressions (pros/cons).

Requirements

But before the actual module is installed, some pre-requirements need to be verified.

1. Icingaweb2 requirements
- Your Icingaweb2 version shouldn't be too old. At least version 2.5 (or newer) is required.
- There are two additional modules needed as dependency: reactbundle and ipl (I'll describe this later)

2. OpenSSL
- You need to have OpenSSL installed.

# apt-get install openssl

3. Database requirements
- The module supports either MySQL or MariaDB
- Make sure the InnoDB variables are the following (in my installation on Ubuntu 16.04 this was already the default):

# mysql -e "show variables where variable_name like 'innodb_file%'"
+--------------------------+-----------+
| Variable_name            | Value     |
+--------------------------+-----------+
| innodb_file_format       | Barracuda |
| innodb_file_format_check | ON        |
| innodb_file_format_max   | Barracuda |
| innodb_file_per_table    | ON        |
+--------------------------+-----------+

# mysql -e "show variables where variable_name like 'innodb_large%'"
+---------------------+-------+
| Variable_name       | Value |
+---------------------+-------+
| innodb_large_prefix | ON    |
+---------------------+-------+

4. PHP requirements
- At least PHP 5.6 is needed, PHP 7.x is recommended
- The additional PHP package "php-gmp" (or "php7.0-gmp") needs to be installed

# apt-get install php7.0-gmp

Installation

If you're already using multiple Icingaweb2 modules, this might be very easy for you. But for me, this was my first additional Icingaweb2 module and some installation points were not very clear to me and had to figure the details out myself (e.g. placing the module into /usr/share/icingaweb2/modules and not into /etc/icingaweb2/modules took me a couple of minutes to figure out).

Clone the repository into /usr/share/icingaweb2/modules:

root@icingaweb2:~# cd /usr/share/icingaweb2/modules/
root@icingaweb2:/usr/share/icingaweb2/modules# git clone https://github.com/Icinga/icingaweb2-module-x509.git

Then rename the newly created directory and give it the correct permissions (optional):

root@icingaweb2:~# mv /usr/share/icingaweb2/modules/icingaweb2-module-x509 /usr/share/icingaweb2/modules/x509
root@icingaweb2:~# chown -R www-data:icingaweb2 /usr/share/icingaweb2/modules/x509/

Now the database needs to be created. The module will use its own database (and therefore database resource).

mysql> CREATE DATABASE x509;
Query OK, 1 row affected (0.00 sec)

mysql> GRANT SELECT, INSERT, UPDATE, DELETE, DROP, CREATE VIEW, INDEX, EXECUTE ON x509.* TO 'icinga'@'localhost' IDENTIFIED BY 'secret';
Query OK, 0 rows affected, 1 warning (0.00 sec)

Then import the schema which is part of the repository you just cloned before:

root@icingaweb2:~# mysql -u root x509 < /usr/share/icingaweb2/modules/x509/etc/schema/mysql.schema.sql

As mentioned before, the x509 module requires two other modules (ipl and reactbundle) as a dependency. They can be installed pretty quickly:

root@icingaweb2:~# REPO="https://github.com/Icinga/icingaweb2-module-ipl" \
&& MODULES_PATH="/usr/share/icingaweb2/modules" \
&& MODULE_VERSION=0.1.1 \
&& mkdir -p "$MODULES_PATH" \
&& git clone ${REPO} "${MODULES_PATH}/ipl" --branch v${MODULE_VERSION}
icingacli module enable ipl

root@icingaweb2:~# REPO="https://github.com/Icinga/icingaweb2-module-reactbundle" \
&& MODULES_PATH="/usr/share/icingaweb2/modules" \
&& MODULE_VERSION=0.4.1 \
&& mkdir -p "$MODULES_PATH" \
&& git clone ${REPO} "${MODULES_PATH}/reactbundle" --branch v${MODULE_VERSION}
icingacli module enable reactbundle

The next steps are taken in the Icingaweb2 user interface. Log in to Icingaweb2 with a privileged user (Administrator role) and enable the x509 module in Configuration -> Modules -> x509. You should also see the other new modules (ipl and reactbundle) already enabled.

As the x509 database is already prepared and ready, we now have to configure it in Icingaweb2 as a resource: Configuration -> Resources -> Create a new resource

Now the x509 module needs to be configured to use this resource in the backend:

Configuration -> Modules -> x509 -> Backend

Select the newly created resource:

So far this was the setup and activation of the x509 module.

Scan jobs

The next step is to create "jobs" which run in the background and scan the network (according to the configured input) for certificates.

But before the first scan job is created, the module's own trust store needs to be filled. When you install the "ca-certificates" packages, you can import these (as Apache user):

www-data@icingaweb2:~$ icingacli x509 import --file /etc/ssl/certs/ca-certificates.crt
Processed 148 X.509 certificates.

Now create the first scan job: Configuration -> Modules -> x509 -> Jobs -> Create a new job

In this example the job named "Subnet253" is created. Every day at 18:00 (6pm) the job should scan through the entire C-Class range 192.168.253.0/24 on port 443.
You can now manually launch the scan job on the cli as apache user:

www-data@icingaweb2:~$ icingacli x509 scan --job Subnet253
openssl verify failed for command openssl verify -CAfile '/tmp/5c3349e6374c4/ca5c3349e637541' '/tmp/5c3349e6374c4/cert5c3349e644d4e' 2>&1: /tmp/5c3349e6374c4/cert5c3349e644d4e: OU = Domain Control Validated, OU = Gandi Standard Wildcard SSL, CN = *.example.com
error 20 at 0 depth lookup:unable to get local issuer certificate

Such a warning can show if either the Root CA issuer is unknown (was not imported into the module's trust store) or if the server was missing a certificate in the chain. This will then also be shown in the list of found certificates in the UI and marked with a red icon:

What the module did in the background is basically a scan in the job's defined IP range. You can manually do the same and verify with the openssl command:

$ openssl s_client -connect 192.168.253.15:443
CONNECTED(00000003)
depth=0 O = Acme Co, CN = Kubernetes Ingress Controller Fake Certificate
verify error:num=20:unable to get local issuer certificate
verify return:1
depth=0 O = Acme Co, CN = Kubernetes Ingress Controller Fake Certificate
verify error:num=21:unable to verify the first certificate
verify return:1
---
Certificate chain
 0 s:/O=Acme Co/CN=Kubernetes Ingress Controller Fake Certificate
   i:/O=Acme Co/CN=Kubernetes Ingress Controller Fake Certificate
---
Server certificate
-----BEGIN CERTIFICATE-----
MIIDbzCCAlegAwIBAgIQNjPMVgsbs/QcapdZeZZ5WDANBgkqhkiG9w0BAQsFADBL
MRAwDgYDVQQKEwdBY21lIENvMTcwNQYDVQQDEy5LdWJlcm5ldGVzIEluZ3Jlc3Mg
Q29udHJvbGxlciBGYWtlIENlcnRpZmljYXRlMB4XDTE4MTIwNDA3NDg1MVoXDTE5
MTIwNDA3NDg1MVowSzEQMA4GA1UEChMHQWNtZSBDbzE3MDUGA1UEAxMuS3ViZXJu
ZXRlcyBJbmdyZXNzIENvbnRyb2xsZXIgRmFrZSBDZXJ0aWZpY2F0ZTCCASIwDQYJ
KoZIhvcNAQEBBQADggEPADCCAQoCggEBAKquGymJpl49Weph8hsusqV4pLOdx6NV
8CCcumTJMSd35VeZUOaHh2mohvkJRaTeXD+QE1VX3vlT2Nt6CCHnM4Q1ldaXdazU
HXGy6XrDDax6GsDR72lpDQ3g6PYwffRwZlVTbISRIhIE0WQLSshjNQ4T29AbOazl
/3DJ2A34BBB3yzWLtMA5HEWDZF8h/RWXJgw2w2gDKq3doY0aYOnpOjjxEOlQIXZ2
GPpv7VHokbGU2f+6myqV9eevLtZy0zKrqmIPualuoDGKhmd0fQv70cA42HZj73Pf
pbbgHb+hSMGAXO1hUkusIfERXTVSxG/OEayrS3MrwGKDL8DrjDZmEeMCAwEAAaNP
ME0wDgYDVR0PAQH/BAQDAgWgMBMGA1UdJQQMMAoGCCsGAQUFBwMBMAwGA1UdEwEB
/wQCMAAwGAYDVR0RBBEwD4INaW5ncmVzcy5sb2NhbDANBgkqhkiG9w0BAQsFAAOC
AQEAeSAKvMz6TpK0MZuNkAwozRE9IQuzrUA77+xgmorYxjxfkX4q1biR6CzQgQx/
uY8LutvKCf5ygnhPJjunhjCCq0OrgWvqj268H8suWxXpErwYP7Nh8Zricn+ALLsq
48mF81tjoOa9FsYUU4hNrkqOMEuPSHIXTr4+xgmdzQjhBrP+tEq9ISwvVX5eQ7E3
BX79v4K3Wb/BFXii1xlPMbLjBAfOCGW9zlCapcil94mfpEHMwqitsgnpurZMhpvH
udQ1nzgXPcFeOCZBZecLnaG1kD2PEL/9zdq9FAB8Bk7iLFloAqFLOjjeTOpv+St6
ehGjRV7Cji4rGDVJSy5pE5GUNA==
-----END CERTIFICATE-----
subject=/O=Acme Co/CN=Kubernetes Ingress Controller Fake Certificate
issuer=/O=Acme Co/CN=Kubernetes Ingress Controller Fake Certificate
---
No client certificate CA names sent
---
SSL handshake has read 1553 bytes and written 421 bytes
---
New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES256-GCM-SHA384
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
SSL-Session:
    Protocol  : TLSv1.2
    Cipher    : ECDHE-RSA-AES256-GCM-SHA384
    Session-ID: AAAEE62F2E75A3FB796C398636E7DA4AA051E7A91D067690C9FCA68DA2BDF0A8
    Session-ID-ctx:
    Master-Key: 1AF1ECB19DC5E830E73BF601878B72F486A710A207315E73168A079D991E2C8D9F72148942028048CFA65B6E2F018E6B
    Key-Arg   : None
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    TLS session ticket lifetime hint: 600 (seconds)
    TLS session ticket:
    0000 - ea 09 28 0c ef 2b 21 a6-22 dd a3 79 2a 14 5e 05   ..(..+!."..y*.^.
    0010 - c1 96 e8 0c dc 11 95 f6-f9 1c aa 18 75 3b ca 61   ............u;.a
    0020 - 4f d8 5e b6 f7 ac 88 fb-f9 e1 6e df 0d 18 3b a2   O.^.......n...;.
    0030 - 4f 54 39 55 07 df e5 d7-81 c0 c5 23 f8 b4 6e 26   OT9U.......#..n&
    0040 - c8 9c 69 53 29 b5 f4 c8-b1 60 fc 63 54 c6 8a e7   ..iS)....`.cT...
    0050 - b7 0d 94 72 dd 26 ce 2a-e7 ed 3d 63 61 2d 77 f8   ...r.&.*..=ca-w.
    0060 - 52 be bc 3d 9e f2 d7 f6-01 c0 c2 ba e0 68 e6 9d   R..=.........h..
    0070 - 72 c4 46 af 07 a7 0e 7c-08 ba fd 67 5f 1e 00 e7   r.F....|...g_...
    0080 - f2 ba 5b c4 0f e2 1c 7e-fb 91 ef e2 b6 9e 12 91   ..[....~........
    0090 - 5b eb bb fd 04 53 4f 47-77 0a 97 f5 ef b7 de 0e   [....SOGw.......
    00a0 - f3 46 04 08 30 5e 08 8d-e6 43 2b 58 f6 da 74 da   .F..0^...C+X..t.

    Start Time: 1546866271
    Timeout   : 300 (sec)
    Verify return code: 21 (unable to verify the first certificate)
---

The x509 module of course found this information on host 192.168.253.15 as well. Because this is an internal Docker host managed by Kubernetes it has an invalid (self signed) certificate from Kubernetes installed. This is nicely shown in the UI, too:

You could now create a cron job which runs the scan task at your wanted time. Or you could also install a SystemD service using the unit file in the repo (see /usr/share/icingaweb2/modules/x509/config/systemd/icinga-x509.service).

SNI certificates

So far so good. The module shows us which certificates are invalid or have a missing chain. But what about SNI certificates?
The module only scans the primary certificate of each IP/host on the configured port (443 in this case).

To use Server Name Indication certificates, the module needs to be told the hostnames (how could it know otherwise). This can be added in the configuration of the module:

Configuration -> Modules -> x509 -> SNI -> Create a new SNI map

So basically you specify for each IP address all the hostnames (comma-separated) which you have configured on that host:

After this you need to run another scan and afterwards all your SNI certificates show up, too.

Remove certificates

But what if a certificate was removed or needs to be deleted?
Well, that as of now is a little problem. The module doesn't delete "old" certificates from the database. It is unfortunately not possible to manually delete a discovered database either. Check out this Github issue where this problem (might) be tackled.

As a workaround you can delete the certificate directly from the database off the table x509_target:

mysql> DELETE FROM x509_target WHERE hostname = "myhostname.example.com";

Where hostname is either the discovered hostname from a scan or the hostname defined in the SNI map.

Pros vs. Cons

The module is relatively new but offers a lot of cool features and graphics. But certain factors (especially SNI certificate monitoring) still requires manual fiddling, more than if you'd build a smart apply rule in Icinga2.

+ TLS Chain verification
+ Certificate verification (self signed vs. validated)
+ Several sorting options
+ Nice overview (dashboard) of all certificates, including CA's
- SNI certificates need to be configured manually as comma-separated hostnames per IP (that's a lot of work if you run huge central reverse proxies)
- Certificates (even falsely discovered ones) cannot be deleted in the UI

Altogether a very handy module even when there's still a lot of room for improvement (which software isn't).

Add a comment

Show form to leave a comment

Comments (newest first)

Claudio from Switzerland wrote on Jan 8th, 2019:

draugas, this article is about the x509 module of the Icingaweb2 user interface. Yes Icingaweb2 only works with Icinga2 and no other monitoring cores/applications. Right now Icinga2 has a very powerful feature (apply rules) which makes it easier and more dynamic to manage large monitoring installations. This is why I "like" Icinga2. But that does not mean that I do not "like" other monitoring solutions. There is nothing bad in Nagios or Naemon (or Thruk or Shinken or Cacti, whatever floats your boat). ;-)
I always make sure the monitoring plugins I create/maintain are independent.

---

draugas from 127.0.0.1 wrote on Jan 8th, 2019:

Why you so like icinga?
What so bad in nagios or naemon?

---