500 Internal Server Error served by Nginx reverse proxy due to worker_connections are not enough

Written by Claudio Kuenzler - 0 comments

Published on August 14th 2020 - Listed in Nginx

---

A Nginx reverse proxy suddenly started to return error 500 on all kinds of sites. The response varied between a real http error (500) and a non-working http communication.

$ curl https://app.example.com -v
* Rebuilt URL to: https://app.example.com/
*   Trying xxx.xxx.xxx.xxx...
* Connected to app.example.com (xxx.xxx.xxx.xxx) port 443 (#0)
* found 127 certificates in /etc/ssl/certs/ca-certificates.crt
* found 520 certificates in /etc/ssl/certs
* ALPN, offering http/1.1
* SSL connection using TLS1.2 / ECDHE_RSA_AES_256_GCM_SHA384
*      server certificate verification OK
*      server certificate status verification SKIPPED
*      common name: *.example.com (matched)
*      server certificate expiration date OK
*      server certificate activation date OK
*      certificate public key: RSA
*      certificate version: #3
*      subject: OU=Domain Control Validated,OU=Gandi Standard Wildcard SSL,CN=*.example.com
*      start date: Thu, 24 Oct 2019 00:00:00 GMT
*      expire date: Thu, 09 Dec 2021 23:59:59 GMT
*      issuer: C=FR,ST=Paris,L=Paris,O=Gandi,CN=Gandi Standard SSL CA 2
*      compression: NULL
* ALPN, server accepted to use http/1.1
> GET / HTTP/1.1
> Host: app.example.com
> User-Agent: curl/7.47.0
> Accept: */*
>
< HTTP/1.1 500 Internal Server Error
< Server: nginx
< Date: Fri, 14 Aug 2020 06:16:54 GMT
< Content-Type: text/html
< Content-Length: 10
< Connection: close
< ETag: "5f362c2c-a"
<
Error 50x
* Closing connection 0

$ curl https://app.example.com -v
* Rebuilt URL to: https://app.example.com/
*   Trying xxx.xxx.xxx.xxx...
* Connected to app.example.com (xxx.xxx.xxx.xxx) port 443 (#0)
* found 127 certificates in /etc/ssl/certs/ca-certificates.crt
* found 520 certificates in /etc/ssl/certs
* ALPN, offering http/1.1
* gnutls_handshake() failed: The TLS connection was non-properly terminated.
* Closing connection 0
curl: (35) gnutls_handshake() failed: The TLS connection was non-properly terminated.

Error logs to the help

Luckily there are error logs available - and the logged error is very much self-explaining:

2020/08/14 08:15:31 [alert] 22158#22158: *469070133 768 worker_connections are not enough while connecting to upstream, client: 54.93.81.216, server: app.example.com, request: "GET /v3/connect/config HTTP/1.1", upstream: "http://127.0.0.1:8105/v3/connect/config", host: "app.example.com"

Increasing worker_connections to fix

worker_connections is a global config option in the events context. On most Nginx installations this can be found in /etc/nginx/nginx.conf.

events {
worker_connections 768;
        # multi_accept on;
}

The default value, here on an Ubuntu 18.04 system, is set to 768 worker_connections. In this case, this reverse proxy received more connections than this and the default worker_connections limit is not high enough.

After setting worker_connections to a higher value and a Nginx restart, the errors were gone and the reverse proxy served the application again.

root@nginx:~# grep worker_connections /etc/nginx/nginx.conf
    worker_connections 1024;

root@nginx:~# systemctl restart nginx

$ curl https://app.example.com -I
HTTP/1.1 200 OK
Server: nginx
Date: Fri, 14 Aug 2020 06:18:45 GMT
Content-Type: application/json
Connection: keep-alive
Vary: Accept-Encoding
Cache-Control: no-cache, no-store, must-revalidate
Expires: Wed 24 Feb 1982 18:42:00 GMT
X-Frame-Options: deny

Monitoring the worker_connections

There are two ways of monitoring the Nginx reverse proxy and therefore "kind of" monitoring its number of worker connections:

• Nginx Status Page (often on /nginx_status or on a manually configured location with stub_status on)
• Using lsof

Add a comment

Show form to leave a comment

Comments (newest first)

No comments yet.