WARNING: Device /dev/xxx not initialized in udev database - shown in rescue chroot environment

Written by Claudio Kuenzler - 0 comments

Published on December 3rd 2020 - Listed in Linux

---

After a recent upgrade from Debian 9 (Stretch) to Debian 10 (Buster), the physical machine failed to boot and landed in grub rescue mode. Although the dist-upgrade ran through without any error, it seems that the updated grub config was not written to the physical hard drives.

Note: This machine uses 2 physical hard drives in a mdadm raid-1. The root (/) partition is set on /dev/md0. The partitions /var and /tmp are logical volumes from the volume group vgsystem.

Time to boot a live Linux and repair the boot loader!

Repairing the boot loader with Knoppix

Once Knoppix was booted from a USB stick, it was time to get that boot loader working again. As this isn't my first rodeo with a non-booting Linux system (see blog posts Debian not booting: ALERT /dev/disk/by-uuid does not exist or Cannot boot Linux Mint 18.1 after running apt-get upgrade) I wasn't losing any sweat (experience pays off, I tell ya!).

For the sake of completeness here are the mount commands I launched within a Terminal in Knoppix:

# vgchange -ay vgsystem
# mount /dev/md0 /mnt
# mount --bind /dev /mnt/dev
# mount --bind /proc /mnt/proc
# mount --bind /sys /mnt/sys
# mount /dev/vgsystem/lvvar /mnt/var
# mount /dev/vgsystem/lvtmp /mnt/tmp
# chroot /mnt /bin/bash

Inside the chroot environment (simulating the real system of this machine), a new initramfs was recreated. But to my big surprise, a lot of WARNING events showed up during the creation of the new initrd:

# mkinitramfs -o /boot/initrd.img-4.19.0-12-amd64
[...]
  WARNING: Device /dev/sda1 not initialized in udev database even after waiting 10000000 microseconds.
  WARNING: Device /dev/ram0 not initialized in udev database even after waiting 10000000 microseconds.
  WARNING: Device /dev/ram1 not initialized in udev database even after waiting 10000000 microseconds.
  WARNING: Device /dev/md0 not initialized in udev database even after waiting 10000000 microseconds.
[...]

Basically mkinitramfs went through each of the devices found in /dev/ (twice) and after a timeout of 10s for each devices continued. Eventually the initramfs was built successfully:

As I have never seen these kind of warnings before, I got curious to find out what the cause is. However I wanted to fix this server first and the next step would be to update grub. But as soon as update-grub was doing its job, the same warnings showed up once more:

Eventually, after another long wait time, this command finished successfully. But why are these warnings showing up?

lvm2 changed device detection to using udev

Research on these WARNINGs led to a topic in the Arch Linux forums. There user leonshaw mentioned that he got the same warnings and they are caused by upstream changes in the lvm2 package. Most likely suspect would be the following change commit:

commit a063d2d123c56c4ccead986625a260df16556b9f
Author: David Teigland <teigland@redhat.com>
Date:   Mon Dec 3 11:22:45 2018 -0600

    devs: use udev info to improve md component detection

    Use udev info to supplement native md component detection.

This change requires the hosts /run mounted into the chroot environment.

Doing the same steps again, with additionally bind mounted /run

With this information in mind, the same mount commands from above with an additional bind mount for /run was run:

# vgchange -ay vgsystem
# mount /dev/md0 /mnt
# mount --bind /dev /mnt/dev
# mount --bind /proc /mnt/proc
# mount --bind /sys /mnt/sys
# mount --bind /run /mnt/run
# mount /dev/vgsystem/lvvar /mnt/var
# mount /dev/vgsystem/lvtmp /mnt/tmp
# chroot /mnt /bin/bash

And update-grub command was run again successfully, this time without the WARNING output:

Note: Some of the logical volumes were blacked out for privacy reasons.

Great, mystery solved. Now I have to keep in mind that in future rescue missions, I should add /run to the bind mounts!

What about the Debian system? How did the story end?

Well, once back in the chroot environment, a simple grub-install on the system disks behind /dev/md0 (detected by the system as SDA and SDD) was run:

# grub-install /dev/sda
# grub-install /dev/sdd

And after a reboot, the new Debian 10 finally booted.

Add a comment

Show form to leave a comment

Comments (newest first)

No comments yet.