How to solve HAProxy running into max open files limit (Proxy reached process FD limit)

Written by Claudio Kuenzler - 1 comments

Published on May 1st 2021 - Listed in HAProxy Systemd Linux

---

Any application with a listener, such as (web) applications, creates a socket for each connection. The number of sockets such an application can create are limited by the so-called "max open files" limit.

In the past (pre Systemd), this problem was solved by adjusting /etc/security/limits.conf. But in the days of applications started by Systemd, this init system applies its own limits. I recommend to read the article "Nginx socket() failed 24: too many open files" to get more details.

HAProxy error: reached process FD limit

HAProxy makes no exception in this case as it is basically also an application with listener(s), creating sockets for each connection. And of course, HAProxy, too, can run into this max open files limit. In such an event, the following log entries can be seen in the HAProxy logs:

Proxy www-out reached process FD limit (maxsock=1023). Please check 'ulimit-n' and restart.

Where "www-out" is the backend name.

The logged error is very helpful actually, because it shows the current limit (1023). This can be verified by checking the limits of the HAProxy process:

root@linux:~# ps auxf|grep haproxy
root     1899651  0.2  0.0  39412 14884 ?        Ss   08:03   0:00 /usr/sbin/haproxy -Ws -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid -S /run/haproxy-master.sock
haproxy  1899654  7.2  0.0 556420 15060 ?        Sl   08:03   0:00  \_ /usr/sbin/haproxy -Ws -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid -S /run/haproxy-master.sock

root@linux:~# cat /proc/1899651/limits | grep "open"
Max open files            1023                 524288               files 

root@linux:~# cat /proc/1899654/limits | grep "open"
Max open files            1023                 524288               files

Both master (PID 1899651) and child (PID 1899654) processes have a "max open files" limit of 1023.

Increase the max open files limit using Systemd unit overwrite

As mentioned at the begin, in the past this would have been solved by modifying /etc/security/limits.conf but with Systemd this is handled differently. One way would be to copy the "original" Systemd unit file of the HAProxy service (/lib/systemd/system/haproxy.service) to /etc/systemd/system/haproxy.service and manually add the LimitNOFILE parameter in it. But a future (security) update of HAProxy could adjust the HAProxy service unit and the update would have no effect.

So the better solution is to create a specific Systemd "sub-unit" configuration. This is done by using a directory named after the service. For HAProxy this is: /etc/systemd/system/haproxy.service.d.

root@linux:~# mkdir /etc/systemd/system/haproxy.service.d

Inside this directory any file with a suffix of .conf will be read by Systemd. Options defined here will append or overwrite existing Systemd unit options.

To increase the limit, create the file /etc/systemd/system/haproxy.service.d/limits.conf with the following content:

root@linux:~# cat /etc/systemd/system/haproxy.service.d/limits.conf
[Service]
LimitNOFILE=500000

Now reload Systemd to read the added configs and then restart HAProxy:

root@linux:~# systemctl daemon-reload
root@linux:~# systemctl restart haproxy

The new limits should now be enabled and this can be verified by reading the limits of the new HAProxy processes:

root@linux:~# ps auxf|grep haproxy
root     1902679  1.0  0.1  84364 46036 ?        Ss   08:06   0:00 /usr/sbin/haproxy -Ws -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid -S /run/haproxy-master.sock
haproxy  1902682  5.3  0.1 614484 44768 ?        Sl   08:06   0:00  \_ /usr/sbin/haproxy -Ws -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid -S /run/haproxy-master.sock

root@linux:~# cat /proc/1902679/limits |grep "open"
Max open files            499999               500000               files    

Add a comment

Show form to leave a comment

Comments (newest first)

Chris from wrote on Feb 17th, 2022:

Thank you for writing this. It was very clear and solved my problem. Kudos!

---