To monitor Citrix from a user endpoint view (=simulating a real user login), we are using Simon Lauger's check_netscaler_gateway monitoring plugin. This has been working very well for the last couple of years - until the plugin started to return a failure one day:
ckadm@mintp ~ $ ./check_netscaler_gateway.pl -H citrix.example.com -u user -p secret -S STORE_ID
NetScaler Gateway CRITICAL - request to https://citrix.example.com/Citrix/STORE_IDWeb/cgi/login failed with HTTP 500
By using the additional -d parameter (for debug), the real reason why the plugin fails is showing:
ckadm@mintp ~ $ ./check_netscaler_gateway.pl -H citrix.example.com -u user -p secret -S STORE_ID -d
$VAR1 = bless( {
'_headers' => bless( {
'client-warning' => 'Internal response',
'content-type' => 'text/plain',
'client-date' => 'Mon, 02 Nov 2020 08:44:09 GMT',
'::std_case' => {
'client-date' => 'Client-Date',
'client-warning' => 'Client-Warning',
'set-cookie2' => 'Set-Cookie2',
'set-cookie' => 'Set-Cookie'
}
}, 'HTTP::Headers' ),
'_rc' => 500,
'_request' => bless( {
'_method' => 'POST',
'_content' => 'login=user&passwd=secret',
'_uri' => bless( do{\(my $o = 'https://citrix.example.com/cgi/login')}, 'URI::https' ),
'_uri_canonical' => $VAR1->{'_request'}{'_uri'},
'_headers' => bless( {
'referer' => 'https://citrix.example.com/vpn/index.html',
'accept' => 'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8',
'content-length' => 33,
'content-type' => 'application/x-www-form-urlencoded',
'user-agent' => 'libwww-perl/6.15'
}, 'HTTP::Headers' )
}, 'HTTP::Request' ),
'_content' => 'Can\'t connect to citrix.example.com:443
SSL connect attempt failed because of handshake problems error:14094410:SSL routines:ssl3_read_bytes:sslv3 alert handshake failure at /usr/share/perl5/LWP/Protocol/http.pm line 47.
',
'_msg' => 'Can\'t connect to citrix.example.com:443'
}, 'HTTP::Response' );
NetScaler Gateway CRITICAL - request to https://citrix.example.com/Citrix/STORE_IDWeb/cgi/login failed with HTTP 500
The relevant part, highlighted in bold, shows a protocol problem (handshake problems error).
But why would a working code suddenly stop working from one day to another? It turns out that on that particular day the plugin started to report CRITICAL, the SSL/TLS versions on the Citrix Netscaler were adjusted. The old SSLv3, TLS1.0 and TLS1.1 protocols were disabled and only TLS1.2 is now accepted. From a security point of view this makes total sense of course - but how can the plugin be fixed?
As the error message hints to a problem in Perl's LWP module, the first guess was that an outdated version of LWP was being used. But when the very same monitoring plugin was executed on a slightly newer OS (Debian Stretch), the plugin ran just fine. A quick comparison between the different package versions revealed that it cannot be a problem of LWP:
| Non-working | Working | |
| Operating System |
Ubuntu 16.04 (xenial) |
Debian 9 (stretch) |
| Perl version |
5.22.1 | 5.24.1 |
| Perl LWP version |
6.06-2 | 6.06-2 |
| Perl SSLeay version |
1.72 |
1.80 |
| OpenSSL version |
1.0.2g |
1.1.0l |
Clearly LWP is the exact same version, however OpenSSL and the Perl module SSLeay (Perl extension for using OpenSSL) differ a lot!
When the plugin / Perl script is executed, SSLeay (basically Openssl) is used in the background. This can be seen with strace, here on Ubuntu xenial:
stat("/etc/perl/Net/SSLeay.pmc", 0x7fff6be9cd90) = -1 ENOENT (No such file or directory)
stat("/etc/perl/Net/SSLeay.pm", 0x7fff6be9ccc0) = -1 ENOENT (No such file or directory)
stat("/usr/local/lib/x86_64-linux-gnu/perl/5.22.1/Net/SSLeay.pmc", 0x7fff6be9cd90) = -1 ENOENT (No such file or directory)
stat("/usr/local/lib/x86_64-linux-gnu/perl/5.22.1/Net/SSLeay.pm", 0x7fff6be9ccc0) = -1 ENOENT (No such file or directory)
stat("/usr/local/share/perl/5.22.1/Net/SSLeay.pmc", 0x7fff6be9cd90) = -1 ENOENT (No such file or directory)
stat("/usr/local/share/perl/5.22.1/Net/SSLeay.pm", 0x7fff6be9ccc0) = -1 ENOENT (No such file or directory)
stat("/usr/lib/x86_64-linux-gnu/perl5/5.22/Net/SSLeay.pmc", 0x7fff6be9cd90) = -1 ENOENT (No such file or directory)
stat("/usr/lib/x86_64-linux-gnu/perl5/5.22/Net/SSLeay.pm", {st_mode=S_IFREG|0644, st_size=52995, ...}) = 0
open("/usr/lib/x86_64-linux-gnu/perl5/5.22/Net/SSLeay.pm", O_RDONLY) = 4
ioctl(4, TCGETS, 0x7fff6be9ca70) = -1 ENOTTY (Inappropriate ioctl for device)
lseek(4, 0, SEEK_CUR) = 0
read(4, "# Net::SSLeay.pm - Perl module f"..., 8192) = 8192
... and later:
read(4, "# Copyright (c) 1997-2007 Graham"..., 8192) = 1324
lseek(4, 1323, SEEK_SET) = 1323
lseek(4, 0, SEEK_CUR) = 1323
close(4) = 0
getuid() = 901
geteuid() = 901
getgid() = 901
getegid() = 901
read(3, "rust path by default, RT#104759\n"..., 8192) = 8192
brk(0x27b3000) = 0x27b3000
getuid() = 901
geteuid() = 901
getgid() = 901
getegid() = 901
read(3, "::INET works. All configuration"..., 8192) = 8192
brk(0x27d4000) = 0x27d4000
read(3, "opened'} = -1;\n\t\t$DEBUG>=1 && DE"..., 8192) = 8192
brk(0x27f5000) = 0x27f5000
brk(0x2816000) = 0x2816000
read(3, "al_ssl_error();\n\t}\n }\n\n $D"..., 8192) = 8192
brk(0x2837000) = 0x2837000
read(3, "ningful\n\t\t last;\n\t\t}\n\n\t\t# ini"..., 8192) = 8192
brk(0x285b000) = 0x285b000
read(3, " => 1,\n\t},\n );\n\n f"..., 8192) = 8192
brk(0x287e000) = 0x287e000
read(3, "ertificate failed because\n# host"..., 8192) = 8192
brk(0x28a0000) = 0x28a0000
brk(0x28c1000) = 0x28c1000
Comparing this to the strace outpout on Debian stretch:
stat("/etc/perl/Net/SSLeay.pmc", 0x7ffdbbb7d080) = -1 ENOENT (No such file or directory)
stat("/etc/perl/Net/SSLeay.pm", 0x7ffdbbb7d080) = -1 ENOENT (No such file or directory)
stat("/usr/local/lib/x86_64-linux-gnu/perl/5.24.1/Net/SSLeay.pmc", 0x7ffdbbb7d080) = -1 ENOENT (No such file or directory)
stat("/usr/local/lib/x86_64-linux-gnu/perl/5.24.1/Net/SSLeay.pm", 0x7ffdbbb7d080) = -1 ENOENT (No such file or directory)
stat("/usr/local/share/perl/5.24.1/Net/SSLeay.pmc", 0x7ffdbbb7d080) = -1 ENOENT (No such file or directory)
stat("/usr/local/share/perl/5.24.1/Net/SSLeay.pm", 0x7ffdbbb7d080) = -1 ENOENT (No such file or directory)
stat("/usr/lib/x86_64-linux-gnu/perl5/5.24/Net/SSLeay.pmc", 0x7ffdbbb7d080) = -1 ENOENT (No such file or directory)
stat("/usr/lib/x86_64-linux-gnu/perl5/5.24/Net/SSLeay.pm", {st_mode=S_IFREG|0644, st_size=52995, ...}) = 0
open("/usr/lib/x86_64-linux-gnu/perl5/5.24/Net/SSLeay.pm", O_RDONLY) = 4
ioctl(4, TCGETS, 0x7ffdbbb7ce50) = -1 ENOTTY (Inappropriate ioctl for device)
lseek(4, 0, SEEK_CUR) = 0
read(4, "# Net::SSLeay.pm - Perl module f"..., 8192) = 8192
And later:
read(4, "# Copyright (c) 1997-2007 Graham"..., 8192) = 1410
lseek(4, 1409, SEEK_SET) = 1409
lseek(4, 0, SEEK_CUR) = 1409
close(4) = 0
getuid() = 0
geteuid() = 0
getgid() = 0
getegid() = 0
read(3, "all_digests();\n\tNet::SSLeay::ran"..., 8192) = 8192
brk(0x55c3c5c37000) = 0x55c3c5c37000
getuid() = 0
geteuid() = 0
getgid() = 0
getegid() = 0
read(3, "NSSL_DIR. Unfortunately it is no"..., 8192) = 8192
brk(0x55c3c5c58000) = 0x55c3c5c58000
brk(0x55c3c5c57000) = 0x55c3c5c57000
read(3, "->{PeerAddr} || $arg_hash->{Peer"..., 8192) = 8192
brk(0x55c3c5c7a000) = 0x55c3c5c7a000
read(3, "\n\t$SSL_OBJECT{$ssl} = [$socket,1"..., 8192) = 8192
brk(0x55c3c5c9b000) = 0x55c3c5c9b000
read(3, "et::SSLeay::peek($ssl,1);\n\tif ( "..., 8192) = 8192
brk(0x55c3c5cbc000) = 0x55c3c5cbc000
brk(0x55c3c5cdd000) = 0x55c3c5cdd000
read(3, "} else {\n *peer_certificates "..., 8192) = 8192
brk(0x55c3c5cfe000) = 0x55c3c5cfe000
read(3, "my ($self,$algo,$cert,$key_only)"..., 8192) = 8192
brk(0x55c3c5d1f000) = 0x55c3c5d1f000
brk(0x55c3c5d40000) = 0x55c3c5d40000
brk(0x55c3c5d3f000) = 0x55c3c5d3f000
read(3, "eaken($handle);\n bless \\$hand"..., 8192) = 8192
brk(0x55c3c5d60000) = 0x55c3c5d60000
read(3, "least one\n\t# buffer was written "..., 8192) = 8192
brk(0x55c3c5d82000) = 0x55c3c5d82000
read(3, "eds Net::SSLeay>=1.56 and OpenSS"..., 8192) = 8192
brk(0x55c3c5da3000) = 0x55c3c5da3000
brk(0x55c3c5da2000) = 0x55c3c5da2000
brk(0x55c3c5dc4000) = 0x55c3c5dc4000
read(3, "lsext_status_cb($ctx,undef);\n\t "..., 8192) = 8192
brk(0x55c3c5de5000) = 0x55c3c5de5000
brk(0x55c3c5de4000) = 0x55c3c5de4000
read(3, "=2 && DEBUG(\"$uri just answered "..., 8192) = 1671
brk(0x55c3c5e05000) = 0x55c3c5e05000
close(3) = 0
On Debian 9 no al_ssl_error occurred.
Obviously the cleanest way is to upgrade the already outdated Operating System (Ubuntu 16.04/xenial). This release will be end of life in April 2021 and should be updated. This way the newer OpenSSL and SSLeay versions will be installed which include compatibility changes for updated protocols and ciphers.
If the OS cannot be upgraded (for whatever reason), the Net::SSLeay Perl module and the openssl libraries should be manually upgraded. See the steps below.
Installing a new version of the Net::SSLeay Perl module using cpan is not enough as it compiles against the already installed OpenSSL libraries (1.0.2g in this case). The same SSL protocol/handshake error would still happen. So first, a new OpenSSL version needs to be downloaded and compiled. It doesn't have to replace the existing openssl version on that machine but the libraries are required.
First download a recent version of OpenSSL (here 1.1.1h):
ckadm@mintp ~/build $ wget https://www.openssl.org/source/openssl-1.1.1h.tar.gz
ckadm@mintp ~/build $ tar -xzf openssl-1.1.1h.tar.gz
ckadm@mintp ~/build $ cd openssl-1.1.1h/
Then create a target directory for the new OpenSSL libraries (default target directory would be /usr/local/lib):
ckadm@mintp ~/build/openssl-1.1.1h $ mkdir ~/build/openssl
This path (/home/ckadm/build/openssl) can now be used as prefix:
ckadm@mintp ~/build/openssl-1.1.1h $ ./config --prefix=/home/ckadm/build/openssl
ckadm@mintp ~/build/openssl-1.1.1h $ make -j24
ckadm@mintp ~/build/openssl-1.1.1h $ make install
Usually this all should work fine and you should have the following directories the target path:
ckadm@mintp ~/build/openssl-1.1.1h $ ll /home/ckadm/build/openssl
total 28
drwxrwxr-x 7 ckadm ckadm 4096 Nov 2 13:27 ./
drwxrwxr-x 4 ckadm ckadm 4096 Nov 2 13:25 ../
drwxrwxr-x 2 ckadm ckadm 4096 Nov 2 13:27 bin/
drwxrwxr-x 3 ckadm ckadm 4096 Nov 2 13:27 include/
drwxrwxr-x 4 ckadm ckadm 4096 Nov 2 13:27 lib/
drwxrwxr-x 4 ckadm ckadm 4096 Nov 2 13:28 share/
drwxrwxr-x 5 ckadm ckadm 4096 Nov 2 13:27 ssl/
The current version of Net::SSLeay (1.88 as of this writing) can easily be downloaded using cpan:
cpan[1]> get Net::SSLeay
cpan[2]> quit
This should download and extract the current release in a .cpan directory.
Note: As an alternative, the release can also be manually downloaded from meta::cpan.
After changing into the source code directory created by cpan, the magic key is to set an environment variable OPENSSL_PREFIX which points to the newer OpenSSL version, followed by perl Makefile.PL.
mintp ~ # cd .cpan/build/Net-SSLeay-1.88-c7vzSa/
mintp Net-SSLeay-1.88-c7vzSa # OPENSSL_PREFIX=/home/ckadm/build/openssl perl Makefile.PL
Do you want to run external tests?
These tests *will* *fail* if you do not have network connectivity. [n]
*** Found OpenSSL-1.1.1h installed in /home/ckadm/build/openssl
*** Be sure to use the same compiler and options to compile your OpenSSL, perl,
and Net::SSLeay. Mixing and matching compilers is not supported.
Checking if your kit is complete...
Looks good
Generating a Unix-style Makefile
Writing Makefile for Net::SSLeay
Writing MYMETA.yml and MYMETA.json
The output already mentions it: OpenSSL 1.1.1h was found.
Now the module can be built and installed (as root - or run sudo make install):
mintp Net-SSLeay-1.88-c7vzSa # make
mintp Net-SSLeay-1.88-c7vzSa # make install
Running Mkbootstrap for Net::SSLeay ()
chmod 644 "SSLeay.bs"
Manifying 2 pod documents
Files found in blib/arch: installing files in blib/lib into architecture dependent library tree
Installing /usr/local/lib/x86_64-linux-gnu/perl/5.22.1/auto/Net/SSLeay/SSLeay.so
Appending installation info to /usr/local/lib/x86_64-linux-gnu/perl/5.22.1/perllocal.pod
Now that the newer Net::SSLeay Perl module was installed (with root privileges) into /usr/local... (which should be part of $PATH), Perl should automatically detect the newer module:
ckadm@mintp ~ $ perl -MNet::SSLeay -le 'print $Net::SSLeay::VERSION'
1.88
So far so good. What about the Perl script, the monitoring plugin check_netscaler_gateway?
ckadm@mintp ~ $ ./check_netscaler_gateway.pl -H citrix.example.com -u user -p secret -S STORE_ID
NetScaler Gateway OK - DESKTOP; Desktop MCP; Desktop Media; Desktop-MCP;
Great! It works again!
Handling https and its protocols has changed a lot in recent years. This is not always obvious. But it becomes a problem when older OpenSSL libraries are used in programs and scripts. A distribution upgrade is often the easiest way to make sure newer ssl/tls protocols and ciphers are accepted. However it is also possible to manually compile and upgrade programs with the newer libraries.
No comments yet.
AWS Android Ansible Apache Apple Atlassian BSD Backup Bash Bluecoat CMS Chef Cloud Coding Consul Containers CouchDB DB DNS Database Databases Docker ELK Elasticsearch Filebeat FreeBSD Galera Git GlusterFS Grafana Graphics HAProxy HTML Hacks Hardware Icinga Influx Internet Java KVM Kibana Kodi Kubernetes LVM LXC Linux Logstash Mac Macintosh Mail MariaDB Minio MongoDB Monitoring Multimedia MySQL NFS Nagios Network Nginx OSSEC OTRS Office OpenSearch PGSQL PHP Perl Personal PostgreSQL Postgres PowerDNS Proxmox Proxy Python Rancher Rant Redis Roundcube SSL Samba Seafile Security Shell SmartOS Solaris Surveillance Systemd TLS Tomcat Ubuntu Unix VMWare VMware Varnish Virtualization Windows Wireless Wordpress Wyse ZFS Zoneminder