LXC 1.1.x is available on Debian Jessie now and update on lxc-start problem

Written by Claudio Kuenzler - 0 comments

Published on March 30th 2016 - Listed in LXC Linux

---

That's what I call a good start into the day: I check my mails and see the following mail from Christian Seiler:

[...] lxc 1:1.1.5-1~bpo8+1 was accepted into jessie-backports today (thanks, backports ftp-masters!) and is already available on the amd64 architecture [...]

As I wrote yesterday, I came across a bug in the "official" 1.0.6 LXC version in Debian Jessie preventing me to start a container after I ran "reboot" in it. Of course I immediately tested 1.1.5 on Jessie now.

I stopped all containers and then upgraded LXC:

root@jessie ~ # apt-get -t jessie-backports install lxc
Reading package lists... Done
Building dependency tree      
Reading state information... Done
The following extra packages will be installed:
  libcgmanager0 libdbus-1-3 liblxc1 libnih-dbus1 libnih1
Suggested packages:
  btrfs-tools lua5.2
Recommended packages:
  dbus apparmor cgmanager dnsmasq-base
The following NEW packages will be installed:
  libcgmanager0 libdbus-1-3 liblxc1 libnih-dbus1 libnih1
The following packages will be upgraded:
  lxc
1 upgraded, 5 newly installed, 0 to remove and 20 not upgraded.
Need to get 1,224 kB of archives.
After this operation, 874 kB of additional disk space will be used.
Do you want to continue? [Y/n] y
Get:1 http://ftp.ch.debian.org/debian/ jessie/main libdbus-1-3 amd64 1.8.20-0+deb8u1 [170 kB]
Get:2 http://ftp.ch.debian.org/debian/ jessie/main libcgmanager0 amd64 0.33-2+deb8u2 [40.7 kB]
Get:3 http://ftp.ch.debian.org/debian/ jessie/main libnih1 amd64 1.0.3-4.3 [127 kB]
Get:4 http://ftp.ch.debian.org/debian/ jessie/main libnih-dbus1 amd64 1.0.3-4.3 [97.1 kB]
Get:5 http://ftp.ch.debian.org/debian/ jessie-backports/main lxc amd64 1:1.1.5-1~bpo8+1 [580 kB]
Get:6 http://ftp.ch.debian.org/debian/ jessie-backports/main liblxc1 amd64 1:1.1.5-1~bpo8+1 [209 kB]
Fetched 1,224 kB in 0s (4,431 kB/s)
Reading changelogs...
apt-listchanges: Mailing root: apt-listchanges: changelogs for jessie
Selecting previously unselected package libdbus-1-3:amd64.
(Reading database ... 25113 files and directories currently installed.)
Preparing to unpack .../libdbus-1-3_1.8.20-0+deb8u1_amd64.deb ...
Unpacking libdbus-1-3:amd64 (1.8.20-0+deb8u1) ...
Selecting previously unselected package libcgmanager0:amd64.
Preparing to unpack .../libcgmanager0_0.33-2+deb8u2_amd64.deb ...
Unpacking libcgmanager0:amd64 (0.33-2+deb8u2) ...
Selecting previously unselected package libnih1.
Preparing to unpack .../libnih1_1.0.3-4.3_amd64.deb ...
Unpacking libnih1 (1.0.3-4.3) ...
Selecting previously unselected package libnih-dbus1.
Preparing to unpack .../libnih-dbus1_1.0.3-4.3_amd64.deb ...
Unpacking libnih-dbus1 (1.0.3-4.3) ...
Preparing to unpack .../lxc_1%3a1.1.5-1~bpo8+1_amd64.deb ...
Unpacking lxc (1:1.1.5-1~bpo8+1) over (1:1.0.6-6+deb8u2) ...
Selecting previously unselected package liblxc1.
Preparing to unpack .../liblxc1_1%3a1.1.5-1~bpo8+1_amd64.deb ...
Unpacking liblxc1 (1:1.1.5-1~bpo8+1) ...
Processing triggers for systemd (215-17+deb8u3) ...
Processing triggers for man-db (2.7.0.2-5) ...
Setting up libdbus-1-3:amd64 (1.8.20-0+deb8u1) ...
Setting up libcgmanager0:amd64 (0.33-2+deb8u2) ...
Setting up libnih1 (1.0.3-4.3) ...
Setting up libnih-dbus1 (1.0.3-4.3) ...
Setting up liblxc1 (1:1.1.5-1~bpo8+1) ...
Setting up lxc (1:1.1.5-1~bpo8+1) ...
Installing new version of config file /etc/apparmor.d/abstractions/lxc/container-base ...
Installing new version of config file /etc/apparmor.d/abstractions/lxc/start-container ...
Installing new version of config file /etc/apparmor.d/lxc/lxc-default-with-nesting ...
Installing new version of config file /etc/init.d/lxc ...
Processing triggers for libc-bin (2.19-18+deb8u3) ...
Processing triggers for systemd (215-17+deb8u3) ...

So far so good. The new version is installed:

root@jessie ~ # dpkg -l | grep lxc
ii  liblxc1          1:1.1.5-1~bpo8+1      amd64        Linux Containers userspace tools (library)
ii  lxc              1:1.1.5-1~bpo8+1      amd64        Linux Containers userspace tools

Then started the container:

root@jessie ~ # lxc-start -n a1 -d && lxc-ls -f | grep a1
lxc-start: utils.c: setproctitle: 1461 Invalid argument - setting cmdline failed
a1     RUNNING  192.168.10.101  -     -       YES 

The output after lxc-start (lxc-start: utils.c: setproctitle: 1461 Invalid argument - setting cmdline failed) is not an error but rather an informative output. This was already discussed in the LXC mailing list back in November 2015 and this should be fixed in the next (1.1.6) release.

Now back to the problem from yesterday again. I attached myself into the container and, like yesterday, ran reboot in it:

root@jessie ~ # lxc-attach -n a1
root@a1 ~ # reboot

Yesterday with LXC 1.0.6 the container was shown in status "STOPPED" and I wasn't able to start it anymore. Let's see if this improved.

The container is like yesterday shown as STOPPED, too.

root@jessie ~ # lxc-ls -f | grep a1
a1     STOPPED  -               -     -       YES  

And worse: A lxc-start didn't work, either! So I was wrong blaming it on the LXC version (although 1.0.6 is very old, still!); there is another problem.
With strace I was able to pin it down to this:

root@jessie ~ # strace -f -s 10000 lxc-start -n a1 -d
[...]
[pid 13913] stat("/dev/vglxc/a1", {st_mode=S_IFBLK|0660, st_rdev=makedev(254, 3), ...}) = 0
[pid 13913] open("/sys/dev/block/254:3/dm/uuid", O_RDONLY) = 9
[pid 13913] fstat(9, {st_mode=S_IFREG|0444, st_size=4096, ...}) = 0
[pid 13913] mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fd521ea0000
[pid 13913] read(9, "LVM-ujPZu604ZotPeJ5Lcz0fofAyVOvLrNqOK203eROfxHf3sEyiJNs0kbSxhmCeSHRi\n", 4096) = 69
[pid 13913] close(9)                    = 0
[pid 13913] munmap(0x7fd521ea0000, 4096) = 0
[pid 13913] access("/etc/filesystems", F_OK) = -1 ENOENT (No such file or directory)
[pid 13913] access("/proc/filesystems", F_OK) = 0
[pid 13913] open("/proc/filesystems", O_RDONLY) = 9
[pid 13913] fstat(9, {st_mode=S_IFREG|0444, st_size=0, ...}) = 0
[pid 13913] mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fd521ea0000
[pid 13913] read(9, "nodev\tsysfs\nnodev\trootfs\nnodev\tramfs\nnodev\tbdev\nnodev\tproc\nnodev\tcgroup\nnodev\tcpuset\nnodev\ttmpfs\nnodev\tdevtmpfs\nnodev\tdebugfs\nnodev\tsecurityfs\nnodev\tsockfs\nnodev\tpipefs\nnodev\tdevpts\nnodev\thugetlbfs\nnodev\tpstore\nnodev\tmqueue\n\text3\n\text2\n\text4\n\txfs\n\tjfs\n\tmsdos\n\tvfat\n\tntfs\n\tminix\n\thfs\n\thfsplus\n\tqnx4\n\tufs\n\tbtrfs\n\tfuseblk\nnodev\tfuse\nnodev\tfusectl\n", 1024) = 344
[pid 13913] mount("/dev/vglxc/a1", "/usr/lib/x86_64-linux-gnu/lxc/rootfs", "ext3", 0, NULL) = -1 EBUSY (Device or resource busy)
[pid 13913] mount("/dev/vglxc/a1", "/usr/lib/x86_64-linux-gnu/lxc/rootfs", "ext2", 0, NULL) = -1 EBUSY (Device or resource busy)
[pid 13913] mount("/dev/vglxc/a1", "/usr/lib/x86_64-linux-gnu/lxc/rootfs", "ext4", 0, NULL) = -1 EBUSY (Device or resource busy)
[pid 13913] mount("/dev/vglxc/a1", "/usr/lib/x86_64-linux-gnu/lxc/rootfs", "xfs", 0, NULL) = -1 EBUSY (Device or resource busy)
[pid 13913] mount("/dev/vglxc/a1", "/usr/lib/x86_64-linux-gnu/lxc/rootfs", "jfs", 0, NULL) = -1 EBUSY (Device or resource busy)
[pid 13913] mount("/dev/vglxc/a1", "/usr/lib/x86_64-linux-gnu/lxc/rootfs", "msdos", 0, NULL) = -1 EBUSY (Device or resource busy)
[pid 13913] mount("/dev/vglxc/a1", "/usr/lib/x86_64-linux-gnu/lxc/rootfs", "vfat", 0, NULL) = -1 EBUSY (Device or resource busy)
[pid 13913] mount("/dev/vglxc/a1", "/usr/lib/x86_64-linux-gnu/lxc/rootfs", "ntfs", 0, NULL) = -1 EBUSY (Device or resource busy)
[pid 13913] mount("/dev/vglxc/a1", "/usr/lib/x86_64-linux-gnu/lxc/rootfs", "minix", 0, NULL) = -1 EBUSY (Device or resource busy)
[pid 13913] mount("/dev/vglxc/a1", "/usr/lib/x86_64-linux-gnu/lxc/rootfs", "hfs", 0, NULL) = -1 EBUSY (Device or resource busy)
[pid 13913] mount("/dev/vglxc/a1", "/usr/lib/x86_64-linux-gnu/lxc/rootfs", "hfsplus", 0, NULL) = -1 EBUSY (Device or resource busy)
[pid 13913] mount("/dev/vglxc/a1", "/usr/lib/x86_64-linux-gnu/lxc/rootfs", "qnx4", 0, NULL) = -1 EBUSY (Device or resource busy)
[pid 13913] mount("/dev/vglxc/a1", "/usr/lib/x86_64-linux-gnu/lxc/rootfs", "ufs", 0, NULL) = -1 EBUSY (Device or resource busy)
[pid 13913] mount("/dev/vglxc/a1", "/usr/lib/x86_64-linux-gnu/lxc/rootfs", "btrfs", 0, NULL) = -1 EBUSY (Device or resource busy)
[pid 13913] mount("/dev/vglxc/a1", "/usr/lib/x86_64-linux-gnu/lxc/rootfs", "fuseblk", 0, NULL) = -1 EBUSY (Device or resource busy)
[pid 13913] read(9, "", 1024)           = 0
[pid 13913] close(9)                    = 0
[pid 13913] munmap(0x7fd521ea0000, 4096) = 0
[pid 13913] write(3, "      lxc-start 1459321346.279 ERROR    bdev - bdev.c:mount_unknown_fs:219 - failed to determine fs type for '/dev/vglxc/a1'\n", 125) = 125
[pid 13913] write(2, "lxc-start: ", 11) = 11
[pid 13913] write(2, "bdev.c: mount_unknown_fs: 219 ", 30) = 30
[pid 13913] write(2, "failed to determine fs type for '/dev/vglxc/a1'", 47) = 47
[pid 13913] write(2, "\n", 1)           = 1
[pid 13913] lstat("/dev", {st_mode=S_IFDIR|0755, st_size=3320, ...}) = 0
[pid 13913] lstat("/dev/vglxc", {st_mode=S_IFDIR|0755, st_size=200, ...}) = 0
[pid 13913] lstat("/dev/vglxc/a1", {st_mode=S_IFLNK|0777, st_size=7, ...}) = 0
[pid 13913] readlink("/dev/vglxc/a1", "../dm-3", 4095) = 7
[pid 13913] lstat("/dev/dm-3", {st_mode=S_IFBLK|0660, st_rdev=makedev(254, 3), ...}) = 0
[pid 13913] access("/dev/dm-3", F_OK)   = 0
[pid 13913] stat("/dev/dm-3", {st_mode=S_IFBLK|0660, st_rdev=makedev(254, 3), ...}) = 0
[...]

For some reason, the logical volume containing the rootfs of this container is still busy. lsof doesn't say anything about it:

root@jessie ~ # lsof /dev/vglxc/a1; echo $?
1

But I know I mounted this LV (and additional volumes for separate partitions) for backup reasons on another location:

root@jessie ~ # cat /proc/mounts | grep vglxc-a1
/dev/mapper/vglxc-a1 /mnt/containerdata/a1 ext4 ro,relatime,data=ordered 0 0
/dev/mapper/vglxc-a1--var /mnt/containerdata/a1/var ext4 rw,relatime,data=ordered 0 0
/dev/mapper/vglxc-a1--opt /mnt/containerdata/a1/opt ext4 rw,relatime,data=ordered 0 0
/dev/mapper/vglxc-a1--tmp /mnt/containerdata/a1/tmp ext4 rw,relatime,data=ordered 0 0
/dev/mapper/vglxc-a1--backup /mnt/containerdata/a1/backup ext4 rw,relatime,data=ordered 0 0

I unmounted all of them:

root@jessie ~ # umount /mnt/containerdata/a1/backup
root@jessie ~ # umount /mnt/containerdata/a1/tmp
root@jessie ~ # umount /mnt/containerdata/a1/opt
root@jessie ~ # umount /mnt/containerdata/a1/var
root@jessie ~ # umount /mnt/containerdata/a1

... and then tried starting the container again:

root@jessie ~ # lxc-start -n a1 -d
lxc-start: utils.c: setproctitle: 1461 Invalid argument - setting cmdline failed

root@jessie ~ # lxc-ls -f | grep a1
a1     RUNNING  192.168.10.101  -     -       YES        

This time it worked! I'm kind of surprised of this because in a kind of similar setup I was able to start containers even though the file systems were mounted. Only difference: That setup was created on Ubuntu servers, not Debian.

Now knowing that the volumes (most probably the additional partitions) shouldn't be mounted after the container was rebooted, I will create a workaround for these partitions, only mounting when the backup should run. And interestingly this only happens if I run "reboot" within the container. If I run "lxc-stop" from the host, the container can be started without a problem.

Anyways, I'm still happy that LXC 1.1.x is now available on Jessie.

Add a comment

Show form to leave a comment

Comments (newest first)

No comments yet.