Not able to mount logical volume used as LXC container rootfs in Proxmox (special device does not exist)

Written by Claudio Kuenzler - 0 comments

Published on September 13th 2024 - Listed in Virtualization Linux Proxmox LXC LVM

---

A new container was created in Proxmox VE. I wanted to migrate container data from another virtualization host to this new Proxmox server by mounting the container's root disk and then use rsync to sync the data. But I ran into a weird problem with the logical volume (used as raw disk for the container).

Container config and its rootfs

Proxmox uses LXC in the background for containers. This makes sense as LXC (a.k.a. System Containers) are perfect for persistent light-weight VMs. I've been using LXC for a decade now, as this blog and the LXC blog tag obviously shows.

The container configuration on Proxmox is slightly different than the LXC default. The Proxmox relevant configuration files for LXC containers are located in /etc/pve/lxc. The following config shows the newly created container and where its rootfs disk can be found (volume group vgssd, logical volume vm-2004-disk-0):

root@proxmox:~# cat /etc/pve/lxc/2004.conf
arch: amd64
cores: 2
features: nesting=1
hostname: mycontainer
memory: 2048
nameserver: 1.1.1.1
net0: name=eth0,bridge=vmbr1,firewall=1,gw=192.168.5.1,hwaddr=BC:24:11:26:9E:43,ip=192.168.5.5/24,type=veth
ostype: debian
rootfs: vgssd:vm-2004-disk-0,size=6G
searchdomain: localdomain.ch
swap: 0
unprivileged: 1

The logical volume (used as rootfs disk) was created by Proxmox and can be seen using lvs:

root@proxmox:~# lvs
  LV             VG    Attr       LSize   Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert
  transfer       vghdd -wi-a-----   1.20t                       
  vm-2004-disk-0 vgssd -wi-------   6.00g

Time to mount the LV and sync data.... 

mount failed: special device /dev/vgssd/vm-2004-disk-0 does not exist

When I tried to mount the Logical Volume, an error message showed up:

root@proxmox:~# mount /dev/vgssd/vm-2004-disk-0 /mnt
mount: /mnt: special device /dev/vgssd/vm-2004-disk-0 does not exist.
       dmesg(1) may have more information after failed mount system call.

First I thought I made a typo in the path. But the lvs output above clearly confirms both VG (vgssd) and LV (vm-2004-disk-0) were correct.

Did Proxmox use the LV as a virtual disk and add partitions into it? But a verification with fdisk also ran into an error:

root@proxmox:~# fdisk -l /dev/vgssd/vm-2004-disk-0
fdisk: cannot open /dev/vgssd/vm-2004-disk-0: No such file or directory

What's going on?!

LV Status: NOT available

The mount error above mentioned to look at dmesg for (potentially) more information. dmesg indeed contained one (somewhat helpful) line:

root@proxmox:~# dmesg|tail
[...]
[63132.597861] /dev/vgssd/vm-2004-disk-0: Can't lookup blockdev

At first I thought "Well, d'uh, I know it did not work". But the keyword blockdev is pretty important here. Let's list all the block devices using dmsetup:

root@proxmox:~# dmsetup ls -o blkdevname
vghdd-transfer    (dm-7)

Only the "transfer" LV is listed. Where is the vgssd-vm-2004-disk-0 block device?

A look at lvdisplay reveals the missing information:

root@proxmox:~# lvdisplay /dev/vgssd/vm-2004-disk-0
  --- Logical volume ---
  LV Path                /dev/vgssd/vm-2004-disk-0
  LV Name                vm-2004-disk-0
  VG Name                vgssd
  LV UUID                arZlTG-ikW0-35Do-ZJtC-JUYp-jQ9U-haDORz
  LV Write Access        read/write
  LV Creation host, time proxmox, 2024-09-13 10:24:44 +0200
  LV Status              NOT available
  LV Size                6.00 GiB
  Current LE             1536
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto

The LV Status shows "NOT available". That's the first time I'm seeing this on a Logical Volume.

I've seen similar situations with Volume Groups (VG) not showing up, because they were not active, but never on a LV. Similar to solve a volume group not showing up (vgchange -ay vgname), a logical volume can be activated:

root@proxmox:~# lvchange -ay /dev/vgssd/vm-2004-disk-0

Right after the lvchange, the LV Status changed to available:

root@proxmox:~# lvdisplay /dev/vgssd/vm-2004-disk-0
  --- Logical volume ---
  LV Path                /dev/vgssd/vm-2004-disk-0
  LV Name                vm-2004-disk-0
  VG Name                vgssd
  LV UUID                arZlTG-ikW0-35Do-ZJtC-JUYp-jQ9U-haDORz
  LV Write Access        read/write
  LV Creation host, time proxmox, 2024-09-13 10:24:44 +0200
  LV Status              available
  # open                 0
  LV Size                6.00 GiB
  Current LE             1536
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     256
  Block device           252:10   

And the "Block device" information is now also present.

Mounting the LV, transferring container data

Without a surprise, the container's rootfs LV could now be mounted and used for data transfer from a different virtualization host:

root@proxmox:~# mount /dev/vgssd/vm-2004-disk-0 /mnt
root@proxmox:~# rm -rf /mnt/*
root@proxmox:~# rsync -a --numeric-ids anotherpve:/ssd-pool/data-ssd/subvol-2004-disk-0/ /mnt/
root@proxmox:~# ls -la /mnt/
total 84
drwxr-xr-x 21 100000 100000 4096 Jul 30 06:46 .
drwxr-xr-x 19 root   root   4096 Aug 15 17:11 ..
drwxr-xr-x  2 100000 100000 4096 Jun 18 06:32 bin
drwxr-xr-x  2 100000 100000 4096 May 13  2019 boot
drwxr-xr-x  2 100000 100000 4096 Dec 10  2008 dev
drwxr-xr-x 87 100000 100000 4096 Jul 30 06:46 etc
drwxr-xr-x  2 100000 100000 4096 May 13  2019 home
drwxr-xr-x 13 100000 100000 4096 Aug 29  2019 lib
drwxr-xr-x  2 100000 100000 4096 Jul  1 06:32 lib64
drwxr-xr-x  2 100000 100000 4096 Jul  8  2019 media
drwxr-xr-x  7 100000 100000 4096 Aug 27  2019 mnt
drwxr-xr-x  2 100000 100000 4096 Jul  8  2019 opt
drwxr-xr-x  2 100000 100000 4096 May 13  2019 proc
drwx------ 11 100000 100000 4096 Apr 23 00:24 root
drwxr-xr-x  3 100000 100000 4096 Jul  8  2019 run
drwxr-xr-x  2 100000 100000 4096 Jul  1 06:32 sbin
drwxr-xr-x  2 100000 100000 4096 Jul  8  2019 srv
drwxr-xr-x  2 100000 100000 4096 May 13  2019 sys
drwxrwxrwt  8 100000 100000 4096 Sep 13 06:13 tmp
drwxr-xr-x 10 100000 100000 4096 Jul  8  2019 usr
drwxr-xr-x 12 100000 100000 4096 Dec 21  2022 var
root@proxmox:~# umount /mnt

Success!

Add a comment

Show form to leave a comment

Comments (newest first)

No comments yet.