Debian Linux failing to boot with dependency error for logical volume using a LVM Thin Pool

Written by Claudio Kuenzler - 0 comments

Published on July 18th 2023 - Listed in Linux LVM

---

A couple of days ago I was contacted to analyze a newly installed Debian 12 (Bookworm) server which was unable to boot. Instead of booting the Kernel until the login prompt, the boot process was stuck mounting the file systems and failed with the error:

[ TIME ] Timed out waiting for device dev-vglxc-34994.device - /dev/vglxc/34994
[DEPEND] Dependency failed for var-lib-lxc-34994-rootfs.mount - /var/lib/lxc/34994/rootfs
[DEPEND] Dependency failed for local-fs.target - Local File Systems.

And the boot processes fell back to the maintenance mode:

Unfortunately the errors (TIME and DEPEND) are not really helpful. From this output alone there's no way of telling what's causing the problem. However the output reveals that the problem seems to be caused while trying to mount the Logical Volume /dev/vglxc/34994 to the path /var/lib/lxc/34994/rootfs. That's at least some information to continue investigating.

Disabling the file system mount in fstab

By disabling the relevant file system entry in /etc/fstab (in maintenance mode or by using a Linux live image and mounting the root partition), the server was able to correctly boot until the login prompt.

Once logged into the server, I was able to dig deeper. The errors seen on the console while Debian was trying to boot could also be seen using journalctl --system:

Jul 17 15:10:18 vst-srv23 systemd[1]: Dependency failed for var-lib-lxc-34994-rootfs.mount - /var/lib/lxc/34994/rootfs.
Jul 17 15:10:18 vst-srv23 systemd[1]: Dependency failed for local-fs.target - Local File Systems.
Jul 17 15:10:18 vst-srv23 systemd[1]: local-fs.target: Job local-fs.target/start failed with result 'dependency'.
Jul 17 15:10:18 vst-srv23 systemd[1]: local-fs.target: Triggering OnFailure= dependencies.
Jul 17 15:10:18 vst-srv23 systemd[1]: var-lib-lxc-34994-rootfs.mount: Job var-lib-lxc-34994-rootfs.mount/start failed with result 'dependency'.

First I wanted to make a file system check on the Logical Volume in question - but surprise! The Logical Volume was not showing up at all inside /dev/. As a matter of fact, nothing from the LVM volume group vglxc was shown. Inside /dev/mapper only the volume groups from the OS setup (vg-root and vg-swap) showed up:

root@vst-srv23:~# ls -la /dev/mapper/
total 0
drwxr-xr-x  2 root root     100 Jul 17 15:25 .
drwxr-xr-x 19 root root    3520 Jul 17 15:25 ..
crw-------  1 root root 10, 236 Jul 17 15:25 control
lrwxrwxrwx  1 root root       7 Jul 17 15:25 vst--srv23--vg-root -> ../dm-0
lrwxrwxrwx  1 root root       7 Jul 17 15:25 vst--srv23--vg-swap_1 -> ../dm-1

Where did the volume group vglxc go?!

Activating the VG vglxc: Missing recommended binary!

When, for whatever reason, a volume group is not showing up, it could be temporarily or administratively disabled. To enable/activate a volume group again, the vgchange command is used:

root@vst-srv23:~# vgchange -ay vglxc
/usr/sbin/thin_check: execvp failed: No such file or directory
WARNING: Check is skipped, please install recommended missing binary /usr/sbin/thin_check!
  2 logical volume(s) in volume group "vglxc" now active

I did not expect this output at all! The warning shows that a recommended binary is missing - the command /usr/sbin/thin_check. It turns out that this volume group is part of a LVM Thin Pool.

Although there were warnings above, the volume group vglxc was enabled and can now be seen in /dev/:

root@vst-srv23:~# ls -la /dev/vg
vga_arbiter  vglxc/

After searching for Debian packages containing that binary /usr/sbin/thin_check it shows that this command is part of the thin-provisioning-tools package. The lvm2 package lists thin-provisioning-tools as a recommended package:

A quick check on that Debian server showed that it was not installed. Let's do this:

root@vst-srv23:~# apt install thin-provisioning-tools
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
The following NEW packages will be installed:
  thin-provisioning-tools
0 upgraded, 1 newly installed, 0 to remove and 0 not upgraded.
Need to get 391 kB of archives.
After this operation, 1438 kB of additional disk space will be used.
Get:1 http://deb.debian.org/debian bookworm/main amd64 thin-provisioning-tools amd64 0.9.0-2 [391 kB]
Fetched 391 kB in 0s (952 kB/s)
Selecting previously unselected package thin-provisioning-tools.
(Reading database ... 43112 files and directories currently installed.)
Preparing to unpack .../thin-provisioning-tools_0.9.0-2_amd64.deb ...
Unpacking thin-provisioning-tools (0.9.0-2) ...
Setting up thin-provisioning-tools (0.9.0-2) ...
Processing triggers for man-db (2.11.2-2) ...
Processing triggers for initramfs-tools (0.142) ...
update-initramfs: Generating /boot/initrd.img-6.1.0-10-amd64

The installation output reveals that this package (thin-provisioning-tools) does have an effect on the boot process, as the initramfs was updated after the package was installed.

After the logical volume mount was enabled in /etc/fstab and a reboot launched, the server successfully booted and mounted the logical volume /dev/vglxc/34994 to the path /var/lib/lxc/34994/rootfs.

TL;DR: LVM Thin Pools require thin-provisioning-tools

When using LVM Thin Pools and auto-mounting these at boot time (usually using /etc/fstab), your system needs the thin-provisioning-tools or the volume group in question cannot be properly activated during boot - causing your system to fail back to the emergency mode.

In Ubuntu Linux, the package thin-provisioning-tools is usually automatically installed as a recommended package of the lvm2 package. But in Debian Linux, the recommended packages are often/by default not installed. In this situation however it turns out it's not just a recommendation, but rather a necessity.

Add a comment

Show form to leave a comment

Comments (newest first)

No comments yet.