When mount stops working in Linux (systemd: mount unit is bound to inactive unit)...

Written by Claudio Kuenzler - 8 comments

Published on September 7th 2021 - Listed in Systemd Linux

---

Today I came across a very weird behaviour involving an otherwise very simple task: Mounting a disk in Linux (Debian 10 Buster to be precise).

For backup purposes I attached an external drive and wanted to mount it on /backup. I made sure this path does not already have a mounted partition or drive and disabled the previous mount in /etc/fstab:

root@buster / # cat /etc/fstab|grep backup
#/dev/mapper/vglxc-backup /backup ext4 defaults 0 0

The additional drive was detected as /dev/sde by the Kernel and the existing partition /dev/sde1 would be used to be mounted at /backup.

root@buster / # fdisk -l /dev/sde
Disk /dev/sde: 931.5 GiB, 1000204886016 bytes, 1953525168 sectors
Disk model: 721010A9E630    
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0xb17d70be

Device     Boot Start        End    Sectors   Size Id Type
/dev/sde1        2048 1953525167 1953523120 931.5G 83 Linux

Time to mount this (ext4 formatted) partition:

root@buster / # mount /dev/sde1 /backup

But interestingly no files showed up - which were supposed to be on the disk:

root@buster / # ll /backup/
total 0

And even more intriguing: df still shows the disk size and usage from the root (/) partition instead of the roughly 900GB partition:

root@buster / # df -h /backup/
Filesystem     Type  Size  Used Avail Use% Mounted on
/dev/md0       ext4   23G  9.7G   12G  45% /

mount: Unit is bound to inactive unit

Taking a closer look at /var/log/syslog, revealed systemd entries immediately unmounting /backup again:

Sep  7 14:21:24 buster systemd[1]: backup.mount: Unit is bound to inactive unit dev-mapper-vglxc\x2dbackup.device. Stopping, too.
Sep  7 14:21:24 buster systemd[1]: Unmounting /backup...
Sep  7 14:21:24 buster systemd[2847]: backup.mount: Succeeded.
Sep  7 14:21:24 buster systemd[1]: backup.mount: Succeeded.
Sep  7 14:21:24 buster systemd[1]: Unmounted /backup.

Seems that Systemd is now also responsible for mounting partitions? 

As this blog article from Leonid Mamchenkov shows, Systemd is indeed behind all this. While mounting, Systemd checks for existing paths on the mount point - and finds the (commented-out!) logical volume entry (/dev/mapper/vglxc-backup). But Systemd does not read /etc/fstab again, it needs to be told to do this. This can be achieved by forcing a Systemd daemon reload:

root@buster / # systemctl daemon-reload

Trying to mount /dev/sde1 again:

root@buster / # mount /dev/sde1 /backup

And finally the mount worked:

root@buster / # df -h /backup
Filesystem     Type  Size  Used Avail Use% Mounted on
/dev/sde1      ext4  916G  187G  730G  21% /backup

The finally mounted /backup mount point now even shows up as its own Systemd unit:

root@buster / # systemctl status backup.mount
- backup.mount - /backup
   Loaded: loaded (/proc/self/mountinfo)
   Active: active (mounted) since Tue 2021-09-07 14:23:10 CEST; 33min ago
    Where: /backup
     What: /dev/sde1

Where does this come from?

Basically Systemd reads /etc/fstab at boot time using its own systemd-fstab-generator. This means changes in /etc/fstab are unknown to Systemd, as the generator only runs once and never again once the system has booted.

systemd-fstab-generator is a generator that translates /etc/fstab (see fstab(5) for details) into native systemd units early at boot and when configuration of the system manager is reloaded.

This also means that only a system reboot or a systemctl daemon-reload would restart systemd-fstab-generator and re-read /etc/fstab.

Add a comment

Show form to leave a comment

Comments (newest first)

Andrea Baldini from wrote on Aug 3rd, 2023:

Thanks a lot! Your post was very useful to understand a fix the problem

---

Anton from Russia wrote on Jul 2nd, 2023:

Thank You VERY match!

---

Aly from wrote on Mar 23rd, 2023:

Well that's really infuriating. mount shows no errors, the drive invisibly unmounts itself again immediately after mount, it wouldn't have occurred to me to try daemon-reload until I checked journalctl (I'd have thought dmesg would be the place for messages like that) and saw the immensely unhelpful error. Thanks for the blog.

---

Gilberto Martins from wrote on Jan 14th, 2023:

Simple and straight to the point, while explaining what caused the issue. I wish many others could write clearly like this! Excellent Job!

In my case for any reason, the UUID has changed. I don't remember whether I've resized the virtual volume or anything else. I updated the /etc/fstab, expecting it to work again. But it didn't. And thanks to this article of yours, it's working again!!!

---

Jian from Toronto Canada wrote on Dec 13th, 2022:

I had the issue after the data was migrated to new SAN. I checked pv, lv, vg without any error, the issue puzzled me a while until I saw your post. Big thanks to you.

---

v2k from wrote on Sep 17th, 2022:

THANK YOU!!!

I thought I'd forgotten how to mount a drive or that there was something wrong with the fs. Strange behavior, people have been known to change and reformat drives...

---

unst4b13 from wrote on Sep 14th, 2022:

This happened to me this morning - made changes and went to mount and nothing. double-checked everything and still nothing. checked /var/log/messages and was like why the heck is systemd unmounting me? didn't know about the /usr/lib/systemd/system-generators only running at boot and daemon-reload. good stuff man thanks.

---

Boom from wrote on Feb 9th, 2022:

Thanks a lot for breaking this down nice and simply for me.

I just came across this, and being relatively new to systemd, I couldn't wrap my head around it.

journalctl was showing lots of messages with an error that it couldn't find a device, and it was becoming a pain to try and get it hooked back up.

As an aside, I also noticed that `\x2d` was being used in place of `-`, and that `-` was replacing `/`, which made the errors A LOT more confusing! :/

---