Most articles I publish here are about technical issues and how to solve them. Sometimes with a real solution, sometimes with a workaround. But I have a feeling that this one will remain a head-scratcher for the years to come...
Of course monitoring HTTPS sites using the monitoring plugin check_http is nothing new and nothing extraordinary. By appending the parameters -S and the optional --sni, a SSL website can be checked:
$ /usr/lib/nagios/plugins/check_http -H www.claudiokuenzler.com -S --sni
HTTP OK: HTTP/1.1 200 OK - 18964 bytes in 0.035 second response time |time=0.035075s;;;0.000000;10.000000 size=18964B;;;0
This also allows to monitor SSL certificates with check_http, but check_ssl_cert is a much better monitoring plugin for this purpose.
However a major problem can occur, when remote sites are running on a server with an older OpenSSL version.
In the following example, check_http from the monitoring-plugins 2.2 package is compiled against OpenSSL 1.1:
ck@monitoring:~$ /usr/lib/nagios/plugins/check_http -V
check_http v2.2 (monitoring-plugins 2.2)
admck@monitoring:~$ ldd /usr/lib/nagios/plugins/check_http
linux-vdso.so.1 (0x00007ffc909ed000)
libssl.so.1.1 => /usr/lib/x86_64-linux-gnu/libssl.so.1.1 (0x00007fc8c56bc000)
libcrypto.so.1.1 => /usr/lib/x86_64-linux-gnu/libcrypto.so.1.1 (0x00007fc8c53e6000)
libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007fc8c53c3000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fc8c51d1000)
libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007fc8c51cb000)
/lib64/ld-linux-x86-64.so.2 (0x00007fc8c5781000)
A HTTP check on a remote server with a lower OpenSSL version (TBV!) then results in the following error:
ck@monitoring:~$ /usr/lib/nagios/plugins/check_http -I 192.168.33.51 -p 7004 -S
CRITICAL - Cannot make SSL connection.
140634064565568:error:14094417:SSL routines:ssl3_read_bytes:sslv3 alert illegal parameter:../ssl/record/rec_layer_s3.c:1543:SSL alert number 47
This error basically means that a SSL handshake could not be established.
Note: In July 2021 I wrote a related article which also handled about the SSL alert number 47 but on a Nginx reverse proxy setup. But back then the problem was related to disabled ciphers - which could be enabled again in /etc/ssl/openssl.cnf. Unfortunately the CipherString workaround does not work in this case.
The first guess from reading the error message above would lead to an assumption that sslv3 was used, right? But even though SSLv3 is indeed enabled by the remote server, so is TLS 1.2 - which should be the preferred protocol. This can be verified with SSL scanners, such as testssl.sh:
ck@monitoring:~/testssl.sh-3.0.7$ ./testssl.sh https://192.168.33.51:7004
###########################################################
testssl.sh 3.0.7 from https://testssl.sh/
This program is free software. Distribution and
modification under GPLv2 permitted.
USAGE w/o ANY WARRANTY. USE IT AT YOUR OWN RISK!
Please file bugs @ https://testssl.sh/bugs/
###########################################################
Using "OpenSSL 1.0.2-chacha (1.0.2k-dev)" [~183 ciphers]
on monitoring:./bin/openssl.Linux.x86_64
(built: "Jan 18 17:12:17 2019", platform: "linux-x86_64")
Start 2022-03-17 11:25:18 -->> 192.168.33.51:7004 <<--
rDNS (192.168.33.51): --
Service detected: HTTP
Testing protocols via sockets except NPN+ALPN
SSLv2 not offered (OK)
SSLv3 not offered (OK)
TLS 1 not offered
TLS 1.1 not offered
TLS 1.2 offered (OK)
TLS 1.3 not offered and downgraded to a weaker protocol
NPN/SPDY not offered
ALPN/HTTP2 not offered
[...]
From the output we can see that testssl is using its own internal OpenSSL version (1.0.2k) to establish a connection to the remote server. The script also detects that three protocols are supported by the remote server: SSLv2, SSLv3 and TLS 1.2.
At the very bottom of the testssl output, a couple of communication simulations are happening. Most interestingly here is the fact, that a connection with OpenSSL 1.1.1d is simulated - and fails:
[...]
Java 8u161 TLSv1.2 ECDHE-RSA-AES256-SHA384, 256 bit ECDH (P-256)
Java 11.0.2 (OpenJDK) TLSv1.2 ECDHE-RSA-AES256-GCM-SHA384, 256 bit ECDH (P-256)
Java 12.0.1 (OpenJDK) TLSv1.2 ECDHE-RSA-AES256-GCM-SHA384, 256 bit ECDH (P-256)
OpenSSL 1.0.2e TLSv1.2 ECDHE-RSA-AES256-GCM-SHA384, 256 bit ECDH (P-256)
OpenSSL 1.1.0l (Debian) TLSv1.2 ECDHE-RSA-AES256-GCM-SHA384, 256 bit ECDH (P-256)
OpenSSL 1.1.1d (Debian) No connection
Thunderbird (68.3) TLSv1.2 ECDHE-RSA-AES128-GCM-SHA256, 256 bit ECDH (P-256)
With that in mind, we can also use testssl to use the system's openssl command, which is 1.1.1f on this Ubuntu 20.04 machine:
ck@monitoring:~/testssl.sh-3.0.7$ ./testssl.sh --openssl=/usr/bin/openssl https://192.168.33.51:7004
No engine or GOST support via engine with your /usr/bin/openssl
###########################################################
testssl.sh 3.0.7 from https://testssl.sh/
This program is free software. Distribution and
modification under GPLv2 permitted.
USAGE w/o ANY WARRANTY. USE IT AT YOUR OWN RISK!
Please file bugs @ https://testssl.sh/bugs/
###########################################################
Using "OpenSSL 1.1.1f 31 Mar 2020" [~79 ciphers]
on monitoring:/usr/bin/openssl
(built: "Mar 9 12:12:45 2022", platform: "debian-amd64")
Start 2022-03-17 11:34:50 -->> 192.168.33.51:7004 <<--
rDNS (192.168.33.51): --
Your OpenSSL cannot connect to 192.168.33.51:7004
The results might look ok but they could be nonsense. Really proceed ? ("yes" to continue) --> yes
Service detected: Couldn't determine what's running on port 7004, assuming no HTTP service => skipping all HTTP checks
testssl now uses OpenSSL 1.1.1f with much less ciphers than before. And it is unable to connect to the remote server.
It does indeed look as if OpenSSL 1.1.1 is incompatible with lower OpenSSL versions. That's very bad news, especially for a monitoring server!
If you've been in the monitoring field as long as I am (17 years meanwhile), you probably have some of the older plugin versions laying around. In this case, I still have an older check_http from the monitoring-plugins package 2.1.2 around:
ck@monitoring:~$ /usr/lib/nagios/plugins/check_http.2.1.2 -V
check_http v2.1.2 (monitoring-plugins 2.1.2)
This older check_http was compiled against OpenSSL 1.0.x:
ck@monitoring:~$ ldd /usr/lib/nagios/plugins/check_http.2.1.2
linux-vdso.so.1 (0x00007ffc39d48000)
libssl.so.1.0.0 => /usr/lib/x86_64-linux-gnu/libssl.so.1.0.0 (0x00007f6712364000)
libcrypto.so.1.0.0 => /usr/lib/x86_64-linux-gnu/libcrypto.so.1.0.0 (0x00007f6711f20000)
libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f6711efd000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f6711d0b000)
libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f6711d05000)
/lib64/ld-linux-x86-64.so.2 (0x00007f67125d7000)
Note: If you've upgraded your Ubuntu from previous versions, you might be in luck and libssl.so.1.0.0 is still on the system. If not, you can take a look at this question on askubuntu and install the older ssl library from a downloaded deb package.
As this check_http version uses the OpenSSL library in version 1.0.x (and not 1.1.x), how does the HTTPS check look like? Let's find out:
ck@monitoring:~$ /usr/lib/nagios/plugins/check_http.2.1.2 -I 192.168.33.51 -p 7004 -S
HTTP WARNING: HTTP/1.1 404 Not Found - 1308 bytes in 0.051 second response time |time=0.051490s;;;0.000000 size=1308B;;;0
It works! Well, with a 404 return code, but the connection could be established!
Affected by this problem, what seems to be an OpenSSL incompatibility, is not only check_http. Basically everything which uses the OpenSSL library in the background, will eventually run into this problem.
This can also be tested with check_ssl_cert. This monitoring plugin uses by default the system's openssl command in the background to obtain and verify certificates. Run against an older web server with OpenSSL 1.0.1f, this results in the following error:
ck@monitoring:~$ /usr/lib/nagios/plugins/check_ssl_cert -H 192.168.33.164 --sni www.example.com
SSL_CERT CRITICAL www.example.com: SSL error: 140715112686912:error:1425F102:SSL routines:ssl_choose_client_version:unsupported protocol:../ssl/statem/statem_lib.c:1941:
But check_ssl_cert has a clever workaround: It allows to specify another openssl command by using the --openssl parameter. Remember testssl from before? This included an openssl binary with version 1.0.2k:
ck@monitoring:~$ /usr/lib/nagios/plugins/check_ssl_cert -H 192.168.33.164 --sni www.example.com --openssl /home/ck/testssl.sh-3.0.7/bin/openssl.Linux.x86_64
SSL_CERT OK - x509 certificate '*.example.com' from 'Gandi Standard SSL CA 2' valid until Feb 4 23:59:59 2023 GMT (expires in 324 days)|days_chain_elem1=324;20;15;; days_chain_elem2=909;20;15;; days_chain_elem3=5786;20;15;;
And - oh wonder - this check works again. Magic!
The verifications with testssl.sh before would indicate that communication with an older OpenSSL (library) version would work:
OpenSSL 1.0.x -> 1.1.1 = works
But if the client runs OpenSSL 1.1.1 and wants to connect to a server with a lower OpenSSL version, this seems to break now:
OpenSSL 1.1.1 -> 1.0.x = no connection
According to the OpenSSL SecurityLevel documentation, the compatibility with older OpenSSL versions can be established by setting the security level to 0:
Level 0
Everything is permitted. This retains compatibility with previous versions of OpenSSL.
But unfortunately this doesn't work.
Right now I'm left pretty baffled and negatively surprised that something so important (openssl backward compatibility) seems to be gone. In an environment where you need to monitor a lot of servers, including legacy systems (which nobody likes, but they do exist) you want to make sure you can continue to monitor your servers. Doesn't matter which OpenSSL they run.
Hence my question to you, dear reader: What did you do in your monitoring environment to tackle this problem? Please leave a comment.
Alright. Meanwhile, a couple of hours after I let out this rant of possible OpenSSL backward incompatibility, focus returned to my mind. Let's get to the bottom of these things.
First, the certificate check (using check_ssl_cert) which failed on a particular host (192.168.33.164). By using testssl.sh I was able to see that this web server only provided SSLv2, SSLv3 and TLSv1 protocols to communicate with:
ck@monitoring:~/testssl.sh-3.0.7# ./testssl.sh https://192.168.33.164
[...]
Testing protocols via sockets except NPN+ALPN
SSLv2 not offered (OK)
SSLv3 offered (NOT ok)
TLS 1 offered (deprecated)
TLS 1.1 not offered
TLS 1.2 not offered and downgraded to a weaker protocol
TLS 1.3 not offered and downgraded to a weaker protocol
NPN/SPDY http/1.1 (advertised)
ALPN/HTTP2 not offered
[...]
The problem with OpenSSL 1.1.1? These protocols are disabled in openssl itself and can't just be enabled in a config file.
After enabling TLS 1.1 and 1.2 on this legacy web server, testssl can now see these protocols, too:
ck@monitoring:~/testssl.sh-3.0.7# ./testssl.sh https://192.168.33.164
[...]
Testing protocols via sockets except NPN+ALPN
SSLv2 not offered (OK)
SSLv3 offered (NOT ok)
TLS 1 offered (deprecated)
TLS 1.1 offered (deprecated)
TLS 1.2 offered (OK)
TLS 1.3 not offered and downgraded to a weaker protocol
NPN/SPDY http/1.1 (advertised)
ALPN/HTTP2 not offered
[...]
The check_ssl_cert command now works with the default openssl command again (without having to specify an older openssl path):
ck@monitoring:~# /usr/lib/nagios/plugins/check_ssl_cert -H 192.168.33.164 --sni www.example.com
SSL_CERT OK - x509 certificate '*.example.com' from 'Gandi Standard SSL CA 2' valid until Feb 4 23:59:59 2023 GMT (expires in 324 days)|days_chain_elem1=324;20;15;; days_chain_elem2=909;20;15;; days_chain_elem3=5786;20;15;;
Back to the initial problem: The check_http failing against a web server, even with TLS 1.2 protocol.
Interestingly, there are two machines of that application. Both are supposed to be set up in the exact same way. Yet the HTTPS checks using check_http 2.2 with OpenSSL 1.1.1 continued to work on the second machine. Why would the check stop working on machine 1 but work on machine 2?
ck@monitoring:~# /usr/lib/nagios/plugins/check_http -I 192.168.33.51 -p 7004 -S
CRITICAL - Cannot make SSL connection.
139827283326272:error:14094417:SSL routines:ssl3_read_bytes:sslv3 alert illegal parameter:../ssl/record/rec_layer_s3.c:1543:SSL alert number 47
ck@monitoring:~# /usr/lib/nagios/plugins/check_http -I 192.168.33.52 -p 7004 -S
HTTP WARNING: HTTP/1.1 404 Not Found - 1308 bytes in 0.049 second response time |time=0.049114s;;;0.000000;10.000000 size=1308B;;;0
We decided to run a cipher scan using nmap on both application servers.
ck@monitoring:~# nmap --script ssl-enum-ciphers -p 7004 192.168.33.51
ck@monitoring:~# nmap --script ssl-enum-ciphers -p 7004 192.168.33.52
Both results showed the same ciphers, however with one difference. The first server with IP ending 51 (with the OpenSSL communication problem) showed a SHA1 signature:
| warnings:
| Key exchange (secp192r1) of lower strength than certificate key
| Weak certificate signature: SHA1
testssl.sh was also used again, this time with a focus on this SHA1 signature. And this SHA1 key can indeed be found in the signature algorithm:
ck@monitoring:~/testssl.sh-3.0.7# ./testssl.sh https://192.168.33.51:7004
[...]
Testing server defaults (Server Hello)
TLS extensions (standard) "EC point formats/#11" "renegotiation info/#65281" "max fragment length/#1"
Session Ticket RFC 5077 hint no -- no lifetime advertised
SSL Session ID support yes
Session Resumption Tickets no, ID: no
TLS clock skew -4 sec from localtime
Signature Algorithm SHA1 with RSA -- besides: users will receive a strong browser WARNING
Server key size RSA 2048 bits
Server key usage --
Server extended key usage --
[...]
Using testssl.sh on the second application server reveals a different signature algorithm with a much newer SHA256 key:
ck@monitoring:~/testssl.sh-3.0.7# ./testssl.sh https://192.168.33.52:7004
[...]
Testing server defaults (Server Hello)
TLS extensions (standard) "EC point formats/#11" "renegotiation info/#65281" "max fragment length/#1"
Session Ticket RFC 5077 hint no -- no lifetime advertised
SSL Session ID support yes
Session Resumption Tickets no, ID: no
TLS clock skew -4 sec from localtime
Signature Algorithm SHA256 with RSA
Server key size RSA 2048 bits
Server key usage --
Server extended key usage --
[...]
SHA1 is considered weak and insecure for many years. Could this be the reason why a communication would not work with OpenSSL 1.1.1?
Once the key was replaced by a newer SHA256 key, testssl.sh now showed this newer key in the signature algorithm:
Testing server defaults (Server Hello)
TLS extensions (standard) "EC point formats/#11" "renegotiation info/#65281" "max fragment length/#1"
Session Ticket RFC 5077 hint no -- no lifetime advertised
SSL Session ID support yes
Session Resumption Tickets no, ID: no
TLS clock skew -4 sec from localtime
Signature Algorithm SHA256 with RSA
Server key size RSA 2048 bits
Server key usage Digital Signature, Key Encipherment
Server extended key usage TLS Web Client Authentication, TLS Web Server Authentication
What about check_http? Does the check work now?
ck@monitoring:~$ /usr/lib/nagios/plugins/check_http -I 192.168.33.51 -p 7004 -S
HTTP WARNING: HTTP/1.1 404 Not Found - 1308 bytes in 0.047 second response time |time=0.046706s;;;0.000000;10.000000 size=1308B;;;0
The HTTP check with check_http now works again!
Both communication problems using the newer check_http with OpenSSL 1.1.1 could be resolved by modifying the target servers (to be monitored) to use newer TLS protocols and by replacing old SHA1 keys.
No comments yet.
AWS Android Ansible Apache Apple Atlassian BSD Backup Bash Bluecoat CMS Chef Cloud Coding Consul Containers CouchDB DB DNS Database Databases Docker ELK Elasticsearch Filebeat FreeBSD Galera Git GlusterFS Grafana Graphics HAProxy HTML Hacks Hardware Icinga Influx Internet Java KVM Kibana Kodi Kubernetes LVM LXC Linux Logstash Mac Macintosh Mail MariaDB Minio MongoDB Monitoring Multimedia MySQL NFS Nagios Network Nginx OSSEC OTRS Office PGSQL PHP Perl Personal PostgreSQL Postgres PowerDNS Proxmox Proxy Python Rancher Rant Redis Roundcube SSL Samba Seafile Security Shell SmartOS Solaris Surveillance Systemd TLS Tomcat Ubuntu Unix VMWare VMware Varnish Virtualization Windows Wireless Wordpress Wyse ZFS Zoneminder