Updating Ubuntu with cross versions on LXC hosts and containers: Watch the gotchas!

Written by Claudio Kuenzler - 0 comments

Published on February 28th 2019 - last updated on April 29th 2019 - Listed in Linux Ubuntu LXC Containers

---

In a previous post not long ago, I wrote that this April 2019 will be the (official) end of the long term support (LTS) of Ubuntu 14.04, codenamed Trusty. I wrote a step by step guide how to upgrade an Ubuntu 14.04 server via 16.04 to 18.04 (see It is 2019, time to upgrade your Ubuntu 14.04 Trusty machines!). This guide covers the basics how to upgrade the Ubuntu distribution and all packages coming from Ubuntu repositories and should work in most setups.

However there are always the more complex scenarios out there. One of such a scenario is one of our LXC environments. The LXC hosts are currently running on Ubuntu 14.04, most of the LXC containers were installed with Ubuntu 16.04.
Note: This is by default not possible due to the different init system of host and containers. However there's a workaround by removing systemd and manually installing upstart-sysv. See article Does an Ubuntu 16.04 (xenial) container run on a 14.04 (trusty) host? for more details.

In the past few days I've been testing which would be the best order to upgrade the Ubuntu versions. Should I start with the containers first, followed by the host? Or should the host have the newest version first and then I upgrade the containers? What about intermediate steps? Will the containers still run on an old 14.04 when the host is on 18.04 and vice versa?

After a couple of upgrades in a TEST environment, I came to the following conclusion and compatibility matrix:

LXC Host	LXC Guest	Compatibility	Notes
14.04	14.04	yes	-
14.04	16.04	no*	System has not been booted with systemd as init system (PID 1). Can't operate. Failed to talk to init daemon.
14.04	18.04	no	System has not been booted with systemd as init system (PID 1). Can't operate. Failed to talk to init daemon.
16.04	14.04	yes	-
16.04	16.04	yes	-
16.04	18.04	yes	-
18.04	14.04	no	upstart init not started, applications could be started manually though.  e.g. /etc/init.d/ssh start https://github.com/lxc/lxc/issues/903
18.04	16.04	yes	-
18.04	18.04	yes	-

* again:  See article Does an Ubuntu 16.04 (xenial) container run on a 14.04 (trusty) host? for a workaround.

Luckliy there is 16.04 as an intermediate version available! It's the version in between being capable of talking correctly to the older upstart and newer systemd init systems!

The worst situations happened when the host staid at 14.04 and all the containers were upgraded but also when the host was upgraded to 18.04 but all the containers remained at 14.04.
Both situations caused the containers to now work correctly. Although they were started by LXC itself, the init system of the containers didn't work anymore due to an incompatibility between Systemd and the previous (Ubuntu-default) init system upstart.

TL;DR

When you need to upgrade an LXC environment (both host and containers need upgrades) and you run multiple versions across host and containers, these are the steps you should consider as your upgrade plan:

1) Upgrade the LXC hosts from 14.04 to 16.04. This will make sure that all containers from 14.04 to 18.04 will start correctly.
2) Upgrade all the containers from 14.04 to 16.04.
3) Upgrade the LXC hosts from 16.04 to 18.04. Containers with 16.04 run on SystemD by default, so they run fine on a host with 18.04.
4) Upgrade all the containers from 16.04 to 18.04.

This will help you reduce the downtimes of the whole LXC environment and therefore your applications.

Updated April 29th 2019:

Here's a handy way to find out under which Linux distribution and which version your containers are running. On the host run os-prober:

root@lxchost:~# os-prober
/dev/mapper/vglxc-dpm--heske01--t:Ubuntu 16.04.2 LTS (16.04):Ubuntu:linux
/dev/mapper/vglxc-inf--jira01--t:Ubuntu 18.04.2 LTS (18.04):Ubuntu1:linux
/dev/mapper/vglxc-inf--ldap01--t:Ubuntu 16.04.1 LTS (16.04):Ubuntu2:linux
/dev/mapper/vglxc-onl--mysql01--test:Ubuntu 18.04.2 LTS (18.04):Ubuntu3:linux
/dev/mapper/vglxc-okapi--app01--test:Ubuntu 18.04.2 LTS (18.04):Ubuntu4:linux
/dev/mapper/vglxc-onl--redis01--t:Ubuntu 16.04.5 LTS (16.04):Ubuntu5:linux
/dev/mapper/vglxc-onlkc--web01--t:Ubuntu 16.04.5 LTS (16.04):Ubuntu6:linux
/dev/mapper/vglxc-shop--web01--t:Ubuntu 18.04.2 LTS (18.04):Ubuntu7:linux
/dev/mapper/vglxc-st--cdb01--t:Ubuntu 16.04.5 LTS (16.04):Ubuntu8:linux
/dev/mapper/vglxc-ver--dks01--t:Ubuntu 16.04.5 LTS (16.04):Ubuntu9:linux

Add a comment

Show form to leave a comment

Comments (newest first)

No comments yet.