Nginx crashes after being reloaded twice (or more) on Ubuntu 20.04

Written by Claudio Kuenzler - 3 comments

Published on May 5th 2021 - Listed in Nginx Linux

---

In the past days, Ubuntu's 16.04 LTS release (Xenial) officially become end of life. This forced a lot of users to switch to the newest LTS 20.04 (Focal) this year - or are still doing migrations right now.

The jump from 16.04 to 20.04 also means a version jump of Nginx: From 1.10.0 on Xenial to 1.18.0 in Focal. We've covered the migration of the Nginx configurations in a separate article: Important Nginx configuration changes after ugprading Ubuntu from 16.04 to 20.04.

However there's something else besides the configuration changes, which is very important: The Nginx reload crash.

Nginx crashes after multiple reloads

Once on Ubuntu 20.04 with Nginx 1.18.0, as you are happily modifying or adding configurations and then reloading, you suddenly notice that HTTP is down and all Nginx processes are gone. What? Why?

This can actually be reproduced by simply reloading Nginx twice (or more) in a row:

root@focal:~# ps aux|grep nginx
root     3356067  0.0  0.1  71600  9788 ?        Ss   15:46   0:00 nginx: master process /usr/sbin/nginx -g daemon on; master_process on;
www-data 3356068  0.0  0.1  73012 15348 ?        S    15:46   0:00 nginx: worker process
www-data 3356069  0.0  0.1  73012 15348 ?        S    15:46   0:00 nginx: worker process
www-data 3356070  0.0  0.1  73012 15348 ?        S    15:46   0:00 nginx: worker process
www-data 3356071  0.0  0.1  73012 15348 ?        S    15:46   0:00 nginx: worker process
www-data 3356072  0.0  0.1  73012 15348 ?        S    15:46   0:00 nginx: worker process
www-data 3356073  0.0  0.1  73012 15348 ?        S    15:46   0:00 nginx: worker process
www-data 3356074  0.0  0.1  73012 15348 ?        S    15:46   0:00 nginx: worker process
www-data 3356075  0.0  0.1  73012 15348 ?        S    15:46   0:00 nginx: worker process
root     3356102  0.0  0.0   8900   676 pts/0    S+   15:46   0:00 grep --color=auto nginx

root@focal:~# service nginx reload

root@focal:~# ps aux|grep nginx
root     3356067  0.2  0.3  81404 28848 ?        Ss   15:46   0:00 nginx: master process /usr/sbin/nginx -g daemon on; master_process on;
www-data 3356266  0.0  0.2  81644 22608 ?        S    15:46   0:00 nginx: worker process
www-data 3356267  0.0  0.2  81644 22608 ?        S    15:46   0:00 nginx: worker process
www-data 3356268  0.0  0.2  81644 22608 ?        S    15:46   0:00 nginx: worker process
www-data 3356269  0.0  0.2  81644 22608 ?        S    15:46   0:00 nginx: worker process
www-data 3356270  0.0  0.2  81644 22608 ?        S    15:46   0:00 nginx: worker process
www-data 3356271  0.0  0.2  81644 22608 ?        S    15:46   0:00 nginx: worker process
www-data 3356272  0.0  0.2  81644 22608 ?        S    15:46   0:00 nginx: worker process
www-data 3356273  0.0  0.2  81644 22608 ?        S    15:46   0:00 nginx: worker process
root     3356296  0.0  0.0  17196  7168 ?        R    15:46   0:00 /usr/bin/pgrep nginx
root     3356309  0.0  0.0   8900   736 pts/0    S+   15:46   0:00 grep --color=auto nginx

root@focal:~# service nginx reload

root@focal:~# ps aux|grep nginx
root     3356355  0.0  0.0   8900   736 pts/0    S+   15:46   0:00 grep --color=auto nginx
root@focal:~#

Note that after the first reload the Nginx processes were still running. After the second reload they were gone.

Additionally to this fact, the following segmentation fault might be logged in /var/log/kern.log right after the second reload:

root@focal:~# tail -f /var/log/kern.log
May  4 15:46:50 focal kernel: [4141537.262674] nginx[3356067]: segfault at 51 ip 00007f6b7d1535c9 sp 00007ffe634334d0 error 4 in libperl.so.5.30.0[7f6b7d0eb000+166000]
May  4 15:46:50 focal kernel: [4141537.262681] Code: 00 0f b6 40 30 49 c1 ed 03 49 29 c5 0f 84 17 01 00 00 48 8b 76 10 48 8b 52 10 4c 8d 3c fe 4c 8d 0c c2 84 c9 0f 84 c7 02 00 00 <49> 83 39 00 0f 85 ad 03 00 00 49 83 c1 08 49 83 ed 01 49 8d 74 1d

Workaround: Disabling the Nginx perl module

From the logged segmentation fault above, the relevant information is that the error happened in libperl.so.5.30.0. This means the Nginx reload triggered something in Perl which returned a (segmentation fault) error and Nginx itself crashed. But what function inside Nginx is using Perl? Actually nothing - however there's a Nginx module for Perl, which is enabled by default:

root@focal:~# ls -l /etc/nginx/modules-enabled/50-mod-http-perl.conf
lrwxrwxrwx 1 root root 53 Mar  4 15:14 /etc/nginx/modules-enabled/50-mod-http-perl.conf -> /usr/share/nginx/modules-available/mod-http-perl.conf

This module (comparable to Apache's mod_perl2) launches Perl in the background and a bug (more on that later in that article) is causing the segmentation error - and the crash.

Most Nginx users actually don't use Perl applications - so for most users it's safe to disable the Perl module:

root@focal:~# rm -f /etc/nginx/modules-enabled/50-mod-http-perl.conf
root@focal:~# systemctl restart nginx

The multiple reload bug is now gone:

root@focal:~# ps auxf|grep nginx
root     3363494  0.0  0.0   8900   736 pts/0    S+   15:55   0:00  |                   \_ grep --color=auto nginx
root     3363349  0.0  0.1  67296  8848 ?        Ss   15:55   0:00 nginx: master process /usr/sbin/nginx -g daemon on; master_process on;
www-data 3363350  0.0  0.1  68724 14704 ?        S    15:55   0:00  \_ nginx: worker process
www-data 3363351  0.0  0.1  68724 14220 ?        S    15:55   0:00  \_ nginx: worker process
www-data 3363352  0.0  0.1  68724 14220 ?        S    15:55   0:00  \_ nginx: worker process
www-data 3363353  0.0  0.1  68724 14220 ?        S    15:55   0:00  \_ nginx: worker process
www-data 3363354  0.0  0.1  68724 14220 ?        S    15:55   0:00  \_ nginx: worker process
www-data 3363355  0.0  0.1  68724 14220 ?        S    15:55   0:00  \_ nginx: worker process
www-data 3363356  0.0  0.1  68724 14220 ?        S    15:55   0:00  \_ nginx: worker process
www-data 3363357  0.0  0.1  68724 14220 ?        S    15:55   0:00  \_ nginx: worker process

root@focal:~# service nginx reload

root@focal:~# ps auxf|grep nginx
root     3363553  0.0  0.0  17196  7036 ?        R    15:56   0:00      \_ /usr/bin/pgrep nginx
root     3363555  0.0  0.0   8900   724 pts/0    S+   15:56   0:00  |                   \_ grep --color=auto nginx
root     3363349  0.2  0.2  74184 23384 ?        Ss   15:55   0:00 nginx: master process /usr/sbin/nginx -g daemon on; master_process on;
www-data 3363531  0.0  0.2  74424 20028 ?        S    15:55   0:00  \_ nginx: worker process
www-data 3363532  0.0  0.2  74424 19648 ?        S    15:55   0:00  \_ nginx: worker process
www-data 3363533  0.0  0.2  74424 19648 ?        S    15:55   0:00  \_ nginx: worker process
www-data 3363534  0.0  0.2  74424 19648 ?        S    15:55   0:00  \_ nginx: worker process
www-data 3363535  0.0  0.2  74424 19648 ?        S    15:55   0:00  \_ nginx: worker process
www-data 3363536  0.0  0.2  74424 19648 ?        S    15:55   0:00  \_ nginx: worker process
www-data 3363537  0.0  0.2  74424 19648 ?        S    15:55   0:00  \_ nginx: worker process
www-data 3363538  0.0  0.2  74424 19648 ?        S    15:55   0:00  \_ nginx: worker process

root@focal:~# service nginx reload

root@focal:~# ps auxf|grep nginx
root     3363618  0.0  0.0   8900   740 pts/0    S+   15:56   0:00  |                   \_ grep --color=auto nginx
root     3363349  0.4  0.2  74208 23424 ?        Ss   15:55   0:00 nginx: master process /usr/sbin/nginx -g daemon on; master_process on;
www-data 3363586  0.0  0.2  74448 20144 ?        S    15:56   0:00  \_ nginx: worker process
www-data 3363587  0.0  0.2  74448 19684 ?        S    15:56   0:00  \_ nginx: worker process
www-data 3363588  0.0  0.2  74448 19684 ?        S    15:56   0:00  \_ nginx: worker process
www-data 3363589  0.0  0.2  74448 19684 ?        S    15:56   0:00  \_ nginx: worker process
www-data 3363590  0.0  0.2  74448 19684 ?        S    15:56   0:00  \_ nginx: worker process
www-data 3363591  0.0  0.2  74448 19684 ?        S    15:56   0:00  \_ nginx: worker process
www-data 3363592  0.0  0.2  74448 19684 ?        S    15:56   0:00  \_ nginx: worker process
www-data 3363593  0.0  0.2  74448 19684 ?        S    15:56   0:00  \_ nginx: worker process

The Nginx processes are still running happily after the second reload. The workaround works!

Bug in Nginx?

But what is actually causing it? A bug in Nginx? Or in Ubuntu itself (e.g. in the Systemd reload mechanism)? Research leads to the obvious bug in Ubuntu's launchpad: Bug #1897561 describes the exact same problems and was confirmed by more than a dozen other Ubuntu users. However a more revealing bug is Nginx bug #1831. This bug was already reported a while ago, in August 2019, and already affected Nginx 1.16. The initial thought was that the Nginx Perl module itself causes the reload crash, however it turns out to be a bug in Perl itself. The bug is fixed in Perl 5.32 and 5.30.2  - however Ubuntu 20.04 ships with the buggy Perl 5.30.0 version.

root@focal:~# dpkg -l|grep "ii  perl"
ii  perl                  5.30.0-9ubuntu0.2    amd64        Larry Wall's Practical Extraction and Report Language
ii  perl-base             5.30.0-9ubuntu0.2    amd64        minimal Perl system
ii  perl-modules-5.26     5.26.1-6ubuntu0.5    all          Core Perl modules
ii  perl-modules-5.30     5.30.0-9ubuntu0.2    all          Core Perl modules

So at the end the ball is in the corner of the Ubuntu package maintainers right now. The Perl package needs to be updated to 5.30.2 or at least contain the patch to fix this bug. Until then use the workaround.

Add a comment

Show form to leave a comment

Comments (newest first)

relative from wrote on Sep 11th, 2022:

Have had this issue for the last couple of months and haven't been able to diagnose it until I read this.

Thanks for your article!

---

ck from Switzerland wrote on May 23rd, 2021:

Hi Francesco. The Perl version stays the same within the LTS Ubuntu version. I was unable to find a repository which contains a newer Perl version for 20.04. Your only way to go would be to self compile and package Perl - that is a big effort however. I hope the Perl package maintainers will upgrade the 5.30 version to 5.30.2 which would contain the fix.

---

Francesco Montanari from wrote on May 23rd, 2021:

Thank you, removing the module worked perfectly!
How to update perl? is there any official apt repository? I didn't find any

---