Unable to create LXC container due to ERROR: Unable to fetch GPG key from keyserver (key server down)

Written by Claudio Kuenzler - 0 comments

Published on June 22nd 2021 - last updated on July 8th 2021 - Listed in Linux LXC Cloud

---

When launching a new LXC container using lxc-create, the following error might show up and container creation fails:

root@host ~ # lxc-create -n ubuntu1 -B lvm --vgname=vglxc --fstype=ext4 --fssize=15G -t download-infiniroot -- -d ubuntu -r focal -a amd64
The cached copy has expired, re-downloading...
Setting up the GPG keyring
ERROR: Unable to fetch GPG key from keyserver

LXC Image verification using GPG key

The reason is that since LXC 3.x, the (suggested) default template is the lxc-download template (/usr/share/lxc/templates/lxc-download). This connects to GPG key servers to verify the downloaded image and prevent a man in the middle (MITM) attack. The connection to the keyserver does not happen via http or https but rather via hkp(s), a different protocol using a different port (tcp/11371). See my notes in a previous articles How to import apt repository key behind outgoing proxy and Setting up a LXC host on Ubuntu 18.04 for additional information related to HKP(S).

But even if the LXC host is able to communicate with the Internet on port tcp/11371, this error can still happen.

Keyserver down

In the background LXC uses the keyservers behind the address pool.sks-keyservers.net. What happened in this particular situation was that no DNS resolving happened for this DNS address:

root@host ~ # dig -t A pool.sks-keyservers.net

; <<>> DiG 9.11.5-P4-5.1+deb10u3-Debian <<>> -t A pool.sks-keyservers.net
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN, id: 50405
;; flags: qr rd ra ad; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 512
;; QUESTION SECTION:
;pool.sks-keyservers.net.    IN    A

;; AUTHORITY SECTION:
sks-keyservers.net.    288    IN    SOA    ns2.kfwebs.net. kf.kfwebs.net. 3210622135 600 14400 172800 600

;; Query time: 3 msec
;; SERVER: 8.8.8.8#53(8.8.8.8)
;; WHEN: Mon Jun 21 20:31:55 CEST 2021
;; MSG SIZE  rcvd: 102

Whatever happened here, DNS caching problem (at Google??!) or DNS misconfiguration of the target domain doesn't really matter. Fact is that at this point in time the key server could not be reached and therefore no GPG key could be downloaded. This leads to no verification of the downloaded image/rootfs and container creation fails.

Where is the key server defined?

The lxc-download template has a hardcoded key server configured:

if [ -z "${DOWNLOAD_KEYSERVER:-}" ]; then
  DOWNLOAD_KEYSERVER="hkp://pool.sks-keyservers.net"

  # Deal with GPG over http proxy
  if [ -n "${http_proxy:-}" ]; then
    DOWNLOAD_KEYSERVER="hkp://p80.pool.sks-keyservers.net:80"
    DOWNLOAD_GPG_PROXY="--keyserver-options http-proxy=\"${http_proxy}\""
  fi
fi

From this code snippet we can also see that a variable ${DOWNLOAD_KEYSERVER} is checked. If it does not exist, the default key server is defined.

This means that an alternative key server can be specified using this variable!

Using a different key server

According to a discussion in the Linuxcontainers community forums, this isn't the first time that the default key servers were not reachable. User simos suggest to switch to the Ubuntu key servers.

With this information, an alternative key server can be defined - right in front of the lxc-create command:

root@host ~ # DOWNLOAD_KEYSERVER="keyserver.ubuntu.com" lxc-create -n ubuntu1 -B lvm --vgname=vglxc --fstype=ext4 --fssize=15G -t download -- -d ubuntu -r focal -a amd64
The cached copy has expired, re-downloading...
Setting up the GPG keyring
Downloading the image index
Downloading the rootfs
Downloading the metadata
The image cache is now ready
Unpacking the rootfs

This time it worked - by using a different GPG key server!

Alternative: Disable download verification

Another (insecure!) alternative is to completely disable the image download verification. There are two ways to do this. Either by using the -no-validate (template) parameter or by modifying the LXC template.

root@host:~# lxc-create -n container1 -B lvm --vgname vglxc --fstype=ext4 --fssize=50G -t download -- --no-validate -d debian -r buster -a amd64

Instead of using the command line parameter, the template (/usr/share/lxc/templates/lxc-download) can also be manually modified by setting the DOWNLOAD_VALIDATE variable to "false". By default this value is set to "true":

root@host:~# grep "^DOWNLOAD_VALIDATE" /usr/share/lxc/templates/lxc-download
DOWNLOAD_VALIDATE="true"

Setting it to false:

root@host:~# sed -i "/^DOWNLOAD_VALIDATE/s/true/false/" /usr/share/lxc/templates/lxc-download

The next lxc-create command adds a warning in the output:

root@host:~# lxc-create -n container1 -B lvm --vgname vglxc --fstype=ext4 --fssize=50G -t download -- -d debian -r buster -a amd64
Downloading the image index
WARNING: Running without gpg validation!
Downloading the rootfs
Downloading the metadata
The image cache is now ready
Unpacking the rootfs
---
You just created a Debian buster amd64 (20210630_05:24) container.

To enable SSH, run: apt install openssh-server
No default root or user password are set by LXC.

Add a comment

Show form to leave a comment

Comments (newest first)

No comments yet.