NRPE in Debian Jessie: Command arguments not working

Written by Claudio Kuenzler - 9 comments

Published on March 16th 2016 - Listed in Nagios Icinga Monitoring Linux

---

When I started monitoring with Nagios back in 2005, I first used check_by_ssh commands. The reason why I switched to NRPE a few years later were mainly these two reasons:

• When the SSH port was being (brute-) attacked, SSH closed connections for further connections. Monitoring was down.
• Under heavy load, the checks executed via SSH timed out very often, causing fault alerts. NRPE times out, too, but in a side by side comparison NRPE was able to "get through" more often.

A few years ago, security problems arose in NRPE when used in combination with command arguments (dont_blame_nrpe setting). But these users which were affected simply didn't care at all about security and let NRPE listening to the whole world. By simply setting the "allowed_hosts" variable and adding some firewall rules (iptables or whatever) the security issue is resolved.

These command arguments are the killer feature of NRPE. Instead of manually setting the values on each host to be monitored, they can be centralized.
The following example shows what I mean.

Imagine a check_disk check on the servers to be monitored. The command is specified in the nrpe config file:

command[check_disk_var]=/usr/lib/nagios/plugins/check_disk -w 20% -c 10% -p /var

Here the values are fixed. 20% space left = warning, critical at 10% left and do this check on partition /var. Of course every server is different. The /var partition may exist, doesn't have to. So for each partition a new command needs to be defined.
And the thresholds may be different as well. 20% warning for a partition with a size of 1GB is more important than if the partition had a size of 1TB.

This is where the command arguments do the perfect job. All fixed values can be replaced by variables:

command[check_disk]=/usr/lib/nagios/plugins/check_disk -w $ARG1$ -c $ARG2$ -W $ARG1$ -K $ARG2$ -p $ARG3$

Here I use the same check_disk plugin, but thresholds (for disk space and inode space) and the partition to be checked are dynamically set by the variables (arguments) received by the NRPE daemon.
On the Nagios/Icinga server side the executed check via the check_nrpe plugin looks like this:

/usr/lib/nagios/plugins/check_nrpe -H target -t 20 -u -p 5666 -c check_disk -a "15%" "5%" "/var"

I've been using this method successfully for years now and you can imagine me scratching my head when I started to monitor a newly installed Debian Jessie server. The NRPE checks didn't work. Enabling the debug log (debug=1) of NRPE revealed:

Mar 16 08:58:02 irhtsrvp01 nrpe[7135]: Connection from monitoringserver port 24011
Mar 16 08:58:02 irhtsrvp01 nrpe[7135]: Host address is in allowed_hosts
Mar 16 08:58:02 irhtsrvp01 nrpe[7135]: Handling the connection...
Mar 16 08:58:02 irhtsrvp01 nrpe[7135]: Error: Request contained command arguments!
Mar 16 08:58:02 irhtsrvp01 nrpe[7135]: Client request was invalid, bailing out...
Mar 16 08:58:02 irhtsrvp01 nrpe[7135]: Connection from �m��z closed.

As I am using the same basic configuration of nrpe.cfg everywhere, I double-checked if I mistakenly disabled the "dont_blame_nrpe" setting. But nope.
Then I came across this Debian bug report (756479). The maintainer of the nagios-nrpe-server package disabled the command arguments by compiling the package without the --enable-command-args option.
While the reason sounds plausible...

"This feature has several security problems and is often used wrong."

... it punishes every other user who used this feature correctly.
It's the same as a political discussion held over and over again in Switzerland: Shall we forbid selling alcohol in public shops after a certain time because this is being abused by (young) people getting drunk on the streets?
A ban would also punish all the other citizens who simply get off work late and have to organize a bottle of wine for friends coming over later or want to get a drink while commuting.
Back to NRPE: According to the maintainer, Alexander Wirt, the decision was to ban the command arguments completely. While I respect his work and decision, I disagree. In my opinion, the package should have been compiled with the command args enabled but disabling it by default in the nrpe.cfg configuration file (actually the same as it was the last years). In the bug report a user (Fabien Coelho) reads my thoughts:

"I know my risks and benefits, and I can make the decision whether to enable arguments or not, you do not have to take this decision for me."

As of now the "workaround" is to recompile the nrpe source package with the --enable-command-args option. The following sites nicely describe how to do this:

• https://menduraa.wordpress.com/2015/10/21/enabling-command-line-arguments-in-nrpe-plugin-debian-jessie/ 
  
• https://fiat-tux.fr/2015/05/19/refaire-le-paquet-debian-nagios-nrpe-server/ (French)

For the sake of completeness, I add this information here, too.

1) Add the deb-src line into your /etc/apt/sources.list file, if it doesn't exist yet:

deb-src http://http.debian.net/debian jessie main

Update the repository list afterwards:

apt-get update

2) Install the build tools and dependencies needed to compile the package:

apt-get install build-essential devscripts debhelper libssl-dev dpatch libwrap0-dev autotools-dev

3) Symlink the openssl library:

ln -s /usr/lib/x86_64-linux-gnu/libssl.so /usr/lib/libssl.so

4) Download the nagios-nrpe-server source package:

apt-get source nagios-nrpe-server

The files will be downloaded into the current directory.

5) Change into the package directory and adapt the debian/rules file:

cd nagios-nrpe-2.15/; vi debian/rules

At the end of the "override_dh_auto_configure" the "--enable-command-args" need to be added:

override_dh_auto_configure:
        ./configure \
                --prefix=/usr \
                --enable-ssl \
                --with-ssl-lib=/usr/lib/$(DEB_HOST_MULTIARCH) \
                --sysconfdir=/etc \
                --localstatedir=/var \
                --libexecdir=/usr/lib/nagios/plugins \
                --libdir=/usr/lib/nagios \
                --enable-command-args

6) Edit the changelog:

dch -i

This command will ask you to enter information what exactly you have done to this package. In my case I entered the following information:

nagios-nrpe (2.15-1.1) UNRELEASED; urgency=medium

  * Non-maintainer upload.
  * Recompile with --enable-command-args

 -- Claudio Kuenzler   Wed, 16 Mar 2016 08:34:28 +0100

7) Compile and create the package:

./configure
make all
dpkg-buildpackage

8) Move one directory up and you will see the newly created files:

cd ..; ll | grep 2.15-1.1
-rw-r--r-- 1 root root   1748 Mar 16 08:37 nagios-nrpe_2.15-1.1_amd64.changes
-rw-r--r-- 1 root root  11580 Mar 16 08:37 nagios-nrpe_2.15-1.1.diff.gz
-rw-r--r-- 1 root root   1104 Mar 16 08:37 nagios-nrpe_2.15-1.1.dsc
-rw-r--r-- 1 root root  19022 Mar 16 08:37 nagios-nrpe-plugin_2.15-1.1_amd64.deb
-rw-r--r-- 1 root root  39268 Mar 16 08:37 nagios-nrpe-server_2.15-1.1_amd64.deb

By looking at the .changes file, we can see the previously entered changelog entry and the different checksums of the changed files:

cat nagios-nrpe_2.15-1.1_amd64.changes
Format: 1.8
Date: Wed, 16 Mar 2016 08:34:28 +0100
Source: nagios-nrpe
Binary: nagios-nrpe-server nagios-nrpe-plugin
Architecture: source amd64
Version: 2.15-1.1
Distribution: UNRELEASED
Urgency: medium
Maintainer: Debian Nagios Maintainer Group
Changed-By: Claudio Kuenzler
Description:
 nagios-nrpe-plugin - Nagios Remote Plugin Executor Plugin
 nagios-nrpe-server - Nagios Remote Plugin Executor Server
Changes:
 nagios-nrpe (2.15-1.1) UNRELEASED; urgency=medium
 .
   * Non-maintainer upload.
   * Recompile with --enable-command-args
Checksums-Sha1:
 dd57d896cc4698a150cb55696f6fd3f9db80ab71 1104 nagios-nrpe_2.15-1.1.dsc
 b0928ac497f4670ca74a5f59df0a8692387b156b 11580 nagios-nrpe_2.15-1.1.diff.gz
 3360ef223a15cc3cb31764d7f058e13f71d5c36f 39268 nagios-nrpe-server_2.15-1.1_amd64.deb
 296a826c17e688f3b36cebb6d299a0d7f0443656 19022 nagios-nrpe-plugin_2.15-1.1_amd64.deb
Checksums-Sha256:
 682d43491235d65130499eb16a5fbcc1424d4664e59217fefa1cec05acc17d70 1104 nagios-nrpe_2.15-1.1.dsc
 a0a73a17d8f6031b098f4eaa9e1482e3feef370fd7325321d8df8276224554a1 11580 nagios-nrpe_2.15-1.1.diff.gz
 67d62603bfa99c5b57ac7c180ed4d77a268d6de59d235ae22028ef77360f8c09 39268 nagios-nrpe-server_2.15-1.1_amd64.deb
 e96c99a46904a2f7986d17624ab9b7e99137bd4c6c582fa9474b32400e5a3457 19022 nagios-nrpe-plugin_2.15-1.1_amd64.deb
Files:
 3ffc7ee400afd760c795ec77ef65ebbc 1104 net optional nagios-nrpe_2.15-1.1.dsc
 1f2d8b45af2d4aed8d8fea2d6ebe1348 11580 net optional nagios-nrpe_2.15-1.1.diff.gz
 4ad15e0dcecedfd3f15a5b91bc4a594f 39268 net optional nagios-nrpe-server_2.15-1.1_amd64.deb
 a6f2e11f0516a4a6c05b4e3832df3281 19022 net optional nagios-nrpe-plugin_2.15-1.1_amd64.deb

9) The deb package can now be installed:

dpkg -i nagios-nrpe-server_2.15-1.1_amd64.deb
Selecting previously unselected package nagios-nrpe-server.
(Reading database ... 26533 files and directories currently installed.)
Preparing to unpack nagios-nrpe-server_2.15-1.1_amd64.deb ...
Unpacking nagios-nrpe-server (2.15-1.1) ...
Setting up nagios-nrpe-server (2.15-1.1) ...
[ ok ] Starting nagios-nrpe: nagios-nrpe.
Processing triggers for man-db (2.7.0.2-5) ...

dpkg -l | grep nrpe
ii  nagios-nrpe-server      2.15-1.1         amd64        Nagios Remote Plugin Executor Server

Now add this deb file and its installation into your configuration management like chef, ansible, puppet,... and deploy it on the Jessie servers.

Update April 22nd 2016: Obviously the same problem happens on the new Ubuntu 16.04 (Xenial) because it is based on Debian Jessie.
You can download recompiled nagios-nrpe-server packages (with command line arguments enabled) for Debian Jessie and Ubuntu Xenial here: http://www.claudiokuenzler.com/downloads/nrpe/ 

Update June 22nd 2017: Command arguments are still disabled in Debian 9 (Stretch). A new guide how to recompile the source package is available on this article: Debian 9 Stretch and Nagios NRPE (command args and SSL compatibility)

Add a comment

Show form to leave a comment

Comments (newest first)

micha from germany wrote on Oct 19th, 2017:

thanks a lot for this great detailed howto !!!

---

Michael from Atlanta, GA wrote on Jun 16th, 2017:

I am 100% with you on this. Setting dont_blame_nrpe=0 by default is sufficient for preventing abuse from non-savvy users. For the rest of us, who understand what we are doing, removing the args parameter from nrpe makes it orders of magnitude less useful and harder to manage. Thanks for posting this.

---

Martin Conzelmann from Deutschland wrote on Jan 26th, 2017:

Claudio,
vielen Dank für diesen Artikel und die Binaries. Das löste mein Problem!
Schöne Grüße,
Martin

---

Max from internet wrote on Nov 27th, 2016:

Finally found your post ! Thanks for the info & the package. Works great.

---

jonci from wrote on Nov 24th, 2016:

Thanks for the article and the repacking.

I can't understand the package's maintenance and I totally agree with Fabien Coelho: I can make the decision whether to enable arguments or not.

Thanks again.

---

VD from Vilnius wrote on Oct 2nd, 2016:

Thanks.

---

ck from Switzerland wrote on Sep 14th, 2016:

Hello Thomas. I just created a 32bit package for Ubuntu xenial and put it on the download site, too.

---

Thomas from wrote on Sep 2nd, 2016:

very helpful, thanks for your work.
But could you provide i386 packages for xenial as well?
That was very great.

---

Hector from Los Angeles wrote on Jun 3rd, 2016:

Awesome post, really helped me keep spinning my wheels trying to troubleshoot this issue on Ubuntu 16.04. Was the first time I set up an internal apt repository to distribute the re-compiled package.

Thanks again!

---