HAProxy in Ubuntu 18.04 Docker image not starting: cannot bind UNIX socket

Written by Claudio Kuenzler - 7 comments

Published on December 4th 2019 - last updated on September 1st 2022 - Listed in Docker HAProxy Linux Kubernetes Containers

---

HAProxy usually works like a charm. In fact HAProxy is (probably) one of the most stable open source software capable of handling thousands of simultaneous connections. Using HAProxy for more than 7 years now, you can imagine my surprise when I was not able to run HAProxy in a Docker/application container.

Umm... what did you do?

Very simple stuff actually. Ubuntu 18.04 was used as base image. In Dockerfile a simple installation of the haproxy package, followed by the RUN of the haproxy process, was added:

FROM ubuntu:18.04
MAINTAINER Claudio Kuenzler

# install packages
RUN apt-get update \
  && apt-get install -y -qq haproxy

CMD ["/usr/sbin/haproxy","-W","-db","-f","/etc/haproxy/haproxy.cfg"]

Once the image was successfully built on Docker Hub, I wanted to deploy the container:

root@host:~# docker run napsty/haproxy:latest
Unable to find image 'napsty/haproxy:latest' locally
latest: Pulling from napsty/haproxy
7ddbc47eeb70: Pull complete
c1bbdc448b72: Pull complete
8c3b70e39044: Pull complete
45d437916d57: Pull complete
8e8788d95679: Pull complete
Digest: sha256:37d36ca920099c8d8288b1207fa7705bc5c169cc8747cf5afd599ab138c316e4
Status: Downloaded newer image for napsty/haproxy:latest
[ALERT] 337/164645 (1) : Starting frontend GLOBAL: cannot bind UNIX socket [/run/haproxy/admin.sock]

The error stating "cannot bind UNIX socket" is unclear without additional information.

Debugging inside the container

Let's get into this container to see what is actually happening:

root@harbor:~# docker run -it napsty/haproxy:devel bash

root@d8e6cf9fa2d5:/# stat /usr/sbin/haproxy
  File: /usr/sbin/haproxy
  Size: 1634568       Blocks: 3200       IO Block: 4096   regular file
Device: fd0eh/64782d    Inode: 682455      Links: 1
Access: (0755/-rwxr-xr-x)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2019-12-04 16:22:23.299177985 +0000
Modify: 2019-12-02 15:38:31.000000000 +0000
Change: 2019-12-04 16:23:27.219793835 +0000
 Birth: -

root@d8e6cf9fa2d5:/# stat /etc/haproxy/haproxy.cfg
  File: /etc/haproxy/haproxy.cfg
  Size: 1276          Blocks: 8          IO Block: 4096   regular file
Device: fd0eh/64782d    Inode: 401588      Links: 1
Access: (0644/-rw-r--r--)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2019-12-04 16:22:23.099176057 +0000
Modify: 2019-10-28 12:01:03.000000000 +0000
Change: 2019-12-04 16:23:27.019791909 +0000
 Birth: -

Both files /usr/sbin/haproxy and /etc/haproxy/haproxy.cfg do exist. But why would HAProxy not start? Launching the RUN command manually should show more:

root@d8e6cf9fa2d5:/# /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg
[ALERT] 337/162548 (13) : Starting frontend GLOBAL: cannot bind UNIX socket [/run/haproxy/admin.sock]

And indeed - HAProxy fails to start because it cannot bind to the Unix socket on /run/haproxy/admin.sock. Interestingly, this directory does not even exist:

root@d8e6cf9fa2d5:/# stat /run/haproxy
stat: cannot stat '/run/haproxy': No such file or directory

What if the directory is added manually?

root@d8e6cf9fa2d5:/# mkdir /run/haproxy
root@d8e6cf9fa2d5:/# /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg
root@d8e6cf9fa2d5:/# ps auxf
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root         1  0.1  0.0  18508  3516 pts/0    Ss   16:23   0:00 bash
haproxy     17  0.0  0.0  54284  1256 ?        Ss   16:27   0:00 /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg
root        18  0.0  0.0  34400  2884 pts/0    R+   16:27   0:00 ps auxf

Ha! HAProxy is running now!

How to solve this situation?

An obvious workaround is to add the creation of the directory already in Dockerfile - at the creation of the image:

FROM ubuntu:18.04
MAINTAINER Claudio Kuenzler

# install packages
RUN apt-get update \
  && apt-get install -y -qq haproxy

# add missing socket path
RUN mkdir /run/haproxy

CMD ["/usr/sbin/haproxy","-W","-db","-f","/etc/haproxy/haproxy.cfg"]

Once the image is built on Docker Hub, a docker pull will download the changed image and the container hopefully starts up this time:

root@host:~# docker pull napsty/haproxy:latest
devel: Pulling from napsty/haproxy
7ddbc47eeb70: Already exists
c1bbdc448b72: Already exists
8c3b70e39044: Already exists
45d437916d57: Already exists
af6d65a824f2: Already exists
cbf99dcc6c4f: Pull complete
b6229eac0350: Pull complete
a5d72fba4c6a: Pull complete
Digest: sha256:c03d7f770fa82ab77bcb2fa021b49300526367b47dc16d2ec9fe89c6be24febd
Status: Downloaded newer image for napsty/haproxy:latest

root@host:~# docker run napsty/haproxy:latest

No errors anymore!

Real solution?

That's the bigger challenge. "Many minds == many opinions"; some may think I'm completely wrong but in my opinion the haproxy package coming from the Ubuntu repositories should be responsible to create this path during the installation (postinst). I created Ubuntu bug #1855140, we'll see how this will be (eventually) solved.

One big question remains though: Why do installations of the haproxy package on "normal" Ubuntu 18.04 work but seem to fail every time when deployed in a Docker container? This can be double-checked in the deb source package.

Analysis of the haproxy source deb package

Note: This package analysis was done after the discussion in the comments (see comments below the article).

After a deb-src repository is added to /etc/apt/sources.list, the haproxy source package can be installed:

root@linux:~# grep src /etc/apt/sources.list
deb-src http://us.archive.ubuntu.com/ubuntu/ bionic main restricted

root@linux:~# apt-get update

root@linux:~# apt-get source haproxy
Reading package lists... Done
NOTICE: 'haproxy' packaging is maintained in the 'Git' version control system at:
https://salsa.debian.org/haproxy-team/haproxy.git
Please use:
git clone https://salsa.debian.org/haproxy-team/haproxy.git
to retrieve the latest (possibly unreleased) updates to the package.
Need to get 2,122 kB of source archives.
Get:1 http://us.archive.ubuntu.com/ubuntu bionic/main haproxy 1.8.8-1 (dsc) [2,280 B]
Get:2 http://us.archive.ubuntu.com/ubuntu bionic/main haproxy 1.8.8-1 (tar) [2,055 kB]
Get:3 http://us.archive.ubuntu.com/ubuntu bionic/main haproxy 1.8.8-1 (diff) [65.2 kB]
Fetched 2,122 kB in 1s (2,280 kB/s)
dpkg-source: info: extracting haproxy in haproxy-1.8.8
dpkg-source: info: unpacking haproxy_1.8.8.orig.tar.gz
dpkg-source: info: unpacking haproxy_1.8.8-1.debian.tar.xz
dpkg-source: info: applying 0002-Use-dpkg-buildflags-to-build-halog.patch
dpkg-source: info: applying haproxy.service-start-after-syslog.patch
dpkg-source: info: applying haproxy.service-add-documentation.patch
dpkg-source: info: applying haproxy.service-use-environment-variables.patch
W: Download is performed unsandboxed as root as file 'haproxy_1.8.8-1.dsc' couldn't be accessed by user '_apt'. - pkgAcquire::Run (13: Permission denied)

Inside the haproxy-1.8.8 folder the structure of the source package can be seen:

root@linux:~# cd haproxy-1.8.8/
root@linux:~/haproxy-1.8.8# ll
total 652
drwxr-xr-x 12 root root   4096 Dec  5 14:15 ./
drwx------  6 root root   4096 Dec  5 14:15 ../
-rw-r--r--  1 root root 490605 Apr 19  2018 CHANGELOG
drwxr-xr-x 18 root root   4096 Apr 19  2018 contrib/
-rw-r--r--  1 root root  41508 Apr 19  2018 CONTRIBUTING
drwxr-xr-x  6 root root   4096 Apr 19  2018 debian/
drwxr-xr-x  5 root root   4096 Apr 19  2018 doc/
drwxr-xr-x  2 root root   4096 Apr 19  2018 ebtree/
drwxr-xr-x  3 root root   4096 Apr 19  2018 examples/
-rw-r--r--  1 root root    788 Apr 19  2018 .gitignore
drwxr-xr-x  6 root root   4096 Apr 19  2018 include/
-rw-r--r--  1 root root   2029 Apr 19  2018 LICENSE
-rw-r--r--  1 root root   3089 Apr 19  2018 MAINTAINERS
-rw-r--r--  1 root root  36682 Apr 19  2018 Makefile
drwxr-xr-x  6 root root   4096 Dec  5 14:15 .pc/
-rw-r--r--  1 root root  15356 Apr 19  2018 README
-rw-r--r--  1 root root   2713 Apr 19  2018 ROADMAP
drwxr-xr-x  2 root root   4096 Apr 19  2018 scripts/
drwxr-xr-x  2 root root   4096 Apr 19  2018 src/
-rw-r--r--  1 root root     14 Apr 19  2018 SUBVERS
drwxr-xr-x  2 root root   4096 Apr 19  2018 tests/
-rw-r--r--  1 root root     24 Apr 19  2018 VERDATE
-rw-r--r--  1 root root      6 Apr 19  2018 VERSION

Building deb packages is not trivial, but if one has gotten into it (for whatever reason) there are a couple of important files to look at. The first file worth to check out is debian/haproxy.dirs:

root@linux:~/haproxy-1.8.8# cat debian/haproxy.dirs
etc/haproxy
etc/haproxy/errors
var/lib/haproxy
var/lib/haproxy/dev

A couple of paths are mentioned here, but the (now famous) /run/haproxy does not appear in that list. However it appears in haproxy.tmpfile:

root@linux:~/haproxy-1.8.8# cat debian/haproxy.tmpfile
d /run/haproxy 2775 haproxy haproxy -

What exactly is this tmpfile and what is its purpose? The last time I personally was involved in more deb package building was during the Debian Wheezy (7) release. This is a pre-systemd Debian release. Might tmpfile be related to systemd? Interestingly this exact information can be found in the package's changelog:

root@linux:~/haproxy-1.8.8# grep -rni tmpfile *
debian/changelog:980:      tmpfiles.d config for systemd.

By taking a closer look at the documentation of dh_systemd_enable it's clearly written:

debian/package.tmpfile
    If this exists, it is installed into usr/lib/tmpfiles.d/package.conf in the package build directory. (The tmpfiles.d mechanism is currently only used by systemd.)

There's no mistake: "currently only used by systemd". And as it is common knowledge: Systemd does not run in a Docker container. 

Further information can also be found on the Ubuntu manpages for tmpfiles.d:

systemd-tmpfiles uses the configuration files from the above directories to describe the creation, cleaning and removal of volatile and temporary files and directories which usually reside in directories such as /run or /tmp.

This sums it up pretty clearly: The /run/haproxy directory is only created by using systemd-tmpfiles, which does not exist in a Docker container. As the haproxy is installed during the Docker image building (which happens inside the Docker container), there is no Systemd available. Hence the directory /run/haproxy is never created.

Add a comment

Show form to leave a comment

Comments (newest first)

Edmond from Ghana wrote on Sep 1st, 2022:

great insight !! thank you

---

Claudio from Switzerland wrote on Jun 11th, 2022:

Hi Javelin,
Add this to your Dockerfile:

# add missing socket path, prepare haproxy.log
RUN mkdir /run/haproxy \
&& chown haproxy:haproxy /run/haproxy \
&& chmod 2775 /run/haproxy

---

Javelin from wrote on Jun 10th, 2022:

Hello Hadret

I'm glad to have found your blog. I'm just starting to learn containers and trying out HAproxy and ran into this issue. It's an excellent write up even a noob like me can understand and follow.

In your last comment, you said that you were able to create the /run/haproxy directory and set permission inside the container.

I'm wondering just how you did that? Can you please post what you did and I'll try to replicate and see if it helps my case.

Thanks!
Jav

---

Hadret from wrote on Dec 5th, 2019:

Ah, nice catch! So the package maintainers never foresaw HAProxy being in environment without systemd available. Nice, thanks for sharing!

---

ck from Switzerland wrote on Dec 5th, 2019:

Hadret, I analysed the source package and it is indeed a problem of the package. It relies on Systemd to create the /run/haproxy directory. I updated the article to show my findings.

---

ck from Switzerland wrote on Dec 5th, 2019:

Hadret, thanks for checking! I am running douzens of HAProxy installations, some of them on Ubuntu 18.04 (LXC or VM's, too) and never had that problem before. So yes, I agree with you - it must have something to do between the package and the fact that this happens in a Docker container. Maybe the haproxy.tmpfile you mentioned is only triggered by systemd? Systemd does not exist within a Docker container and maybe this causes the problem? I will have to look at the deb src package I guess.

---

Hadret from wrote on Dec 5th, 2019:

I dug a bit out of sheer curiosity. It's not a problem with the package -- it's a problem with Docker container. Not sure where is it exactly, but the package does handle /run/haproxy folder creation. In the debian folder of the package you can find haproxy.tmpfile that is responsible for creating it:

d /run/haproxy 2775 haproxy haproxy -

What's more, if you install haproxy package on a VM (default/fresh/clean OS), the directory is being created successfully:

root@haproxy:~# ls -ld /run/haproxy
drwxrwsr-x 2 haproxy haproxy 60 Dec 5 13:26 /run/haproxy

It even matches the setgid (2755) setting. My bet is on Docker not handling that well on the package installation. For whatever reason -- I was able to create the directory and give it ownership and appropriate permissions inside the container.

---