NRPE: Error - Could not complete SSL handshake after check_nrpe was upgraded from 2.x to 3.x

Written by Claudio Kuenzler - 3 comments

Published on March 25th 2021 - last updated on May 2nd 2021 - Listed in Monitoring Linux

---

After a monitoring server, on which the nagios-plugin check_nrpe runs, was upgraded from Ubuntu 16.04 (xenial) to 18.04 (bionic), the check_nrpe plugin was upgraded, too.

Incompatible versions: nrpe 2.x <-> nrpe 3.x

On a Ubuntu 16.04, the nagios-nrpe-plugin package is installed in version 2.15. On Ubuntu 18.04, the plugin is upgraded to 3.2.1:

root@monitoring:~# dpkg -l|grep nrpe-plugin
ii  nagios-nrpe-plugin       3.2.1-1ubuntu1     amd64        Nagios Remote Plugin Executor Plugin

From that moment on, a couple of alerts appeared in our monitoring system (running Icinga 2). Target hosts still running on Ubuntu Xenial (and therefore running nagios-nrpe-server 2.15) would now return the following error:

root@monitoring:~# /usr/lib/nagios/plugins/check_nrpe -H xenial -c check_command
CHECK_NRPE: (ssl_err != 5) Error - Could not complete SSL handshake with xenial: 1

Obviously there is an (SSL) incompatibility between these two versions. Something similar also happened in the "parent" distribution, Debian (see Debian 9 Strech and Nagios NRPE command args and SSL compatibility) but this was fixed in the Debian 9 packages.

Update May 2nd 2021:

The same problem now happens in Debian 10 Buster, when NRPE was upgraded from 3.0.1-3+deb9u1 to 3.2.1-2.

Compile nrpe (server) from source

Luckily nrpe is a very leightweight application, and it's easy to compile the binary from source.

For the sake of completeness, here are the steps to manually compile nrpe from source.

ck@xenial:~$ sudo apt-get update
ck@xenial:~$ sudo apt-get install -y build-essential autoconf automake gcc libc6 libmcrypt-dev make libssl-dev wget openssl
ck@xenial:~$ wget --no-check-certificate -O nrpe.tar.gz https://github.com/NagiosEnterprises/nrpe/archive/nrpe-4.0.3.tar.gz
ck@xenial:~$ tar xzf nrpe.tar.gz
ck@xenial:~$ cd nrpe-nrpe-4.0.3/
ck@xenial:~$ ./configure --prefix=/usr --sysconfdir=/etc --libdir=/usr/lib/nagios --libexecdir=/usr/lib/nagios/plugins --localstatedir=/var --enable-ssl --with-ssl-lib=/usr/lib/$(DEB_HOST_MULTIARCH) --with-piddir=/var/run/nagios --enable-command-args
ck@xenial:~$ make nrpe

If all of these commands ran through without any error, this should result in a binary file called nrpe in the src directory:

ck@xenial:~/src/nrpe-nrpe-4.0.3$ ll src/nrpe
-rwxr-xr-x 1 root root 254552 Mar 25 08:50 src/nrpe*

Replace the nrpe (server) binary on the affected servers

This binary can now be placed on the Ubuntu 16.04 servers still running nagios-nrpe-server 2.15. Simply stop nagios-nrpe-server, create a backup of /usr/sbin/nrpe, replace /usr/sbin/nrpe with the new binary and finally start nagios-nrpe-server again:

root@xenial:~# wget https://www.claudiokuenzler.com/downloads/nrpe/nrpe-4.0.3-xenial
root@xenial:~# chmod 755 nrpe-4.0.3-xenial
root@xenial:~# cp /usr/sbin/nrpe{,.backup}
root@xenial:~# systemctl stop nagios-nrpe-server
root@xenial:~# cp nrpe-4.0.3-xenial /usr/sbin/nrpe
cp: overwrite '/usr/sbin/nrpe'? y
root@xenial:~# systemctl start nagios-nrpe-server

As you can see, the nrpe binary for Ubuntu Xenial is prepared and available publicly. But for security reasons, you should compile your own binary.

Note: A way better solution would be to create a deb package for the new nrpe version for Xenial but the effort would have been much higher and all my Xenial systems will be either upgraded or shut down this year.

Does it work now?

Right after replacing the binary and restarting nagios-nrpe-server the checks recovered. A manual check confirms that the compatibility between the plugin (3.2.1) and the server (4.0.3) is working:

root@monitoring:~# /usr/lib/nagios/plugins/check_nrpe -H xenial
NRPE v4.0.3

root@monitoring:~# /usr/lib/nagios/plugins/check_nrpe -H xenial -c check_load -a "10,9,8" "20,15,12"
OK - load average: 0.07, 0.02, 0.00|load1=0.070;10.000;20.000;0; load5=0.020;9.000;15.000;0; load15=0.000;8.000;12.000;0;

Update: Workaround using newer check_nrpe with NRPE 2 daemon

Updated May 2nd 2021

The same error (CHECK_NRPE: (ssl_err != 5) Error - Could not complete SSL handshake) is now also seen after upgrading Icinga 2's OS from Debian Stretch to Buster. If the target system is still using NRPE 2, the following workaround can be applied: Switch to non-ssl connection.

On the target, where NRPE runs as daemon, adjust the nrpe startup script (e.g. /etc/init.d/nagios-nrpe-server) and add the command parameter -n to it:

[...]
case "$1" in
  start)
    if [ "$INETD" = 1 ]; then
        exit 1
    fi
    log_daemon_msg "Starting $DESC" "$NAME"
    start_daemon $NICENESS $DAEMON -c $CONFIG -d -n $DAEMON_OPTS
    log_end_msg $?
    ;;
[...]

Then restart nrpe on the target.

On the monitoring server, where check_nrpe plugin is launched, use the -n parameter in the command line, too.

If you are using Icinga 2 with apply rules, the following adjustment (Handle old NRPE versions) can help you:

root@icinga2:~# cat /etc/icinga2/zones.d/global-templates/applyrules/cpuload.conf
apply Service "CPU Load" {

  display_name = "CPU Load"
  import "generic-service"
  check_command = "nrpe"
  vars.nrpe_command = "check_load"

  # Handle old NRPE versions (SSL incompatibility)
  if (host.vars.distro.version == "6") {
    vars.nrpe_no_ssl = true
        vars.nrpe_version_2 = true
  }

  assign where host.address && host.vars.os == "Linux"
  ignore where host.vars.applyignore.cpuload == true
}

Of course this requires, that the actual host(s) still running an old NRPE version have the custom variable "vars.distro.version" set.

Disabling the encryption is really only considered a workaround. You should make sure that:

• You don't pass any passwords as NRPE arguments
• Firewall is enabled and only your monitoring server is allowed to connect to tcp/5666
  

Add a comment

Show form to leave a comment

Comments (newest first)

Christian from wrote on Mar 9th, 2022:

Hi,

correction, we see this message on Debian Bullseye hosts and on the Buster hosts, but the checks work nonetheless. It's just regularly that some of the checks fail, but when checked again it works..
There is a Ubuntu bug ticket about that, but couldn't find anything for Debian or NRPE directly.
https://bugs.launchpad.net/ubuntu/+source/nagios-nrpe/+bug/1888184

---

Christian from wrote on Mar 9th, 2022:

Hi,
we are currently seeing the same problem with NRPE 3.2.1-2 on server side (running Debian Buster) and 4.0.3-1 on client side (running Debian Bullseye).

---

Douglas Danger Manley from wrote on Jul 13th, 2021:

Robbert Müller has a PPA for this exact problem; his package will deploy the same old NRPE version 2.15, but with the DH key set to 2048, and it's working perfectly for us until we can get all of our old Ubuntu 16.04 machines upgraded.

https://launchpad.net/~mjrider/+archive/ubuntu/nagios-nrpe/+packages

Note that I couldn't get it to work *as* a PPA, but I downloaded the nagios-nrpe-server_2.15-1ubuntu1.1+dd1_amd64 ".deb" file and installed that directly and it worked fine.

---