Error during Ubuntu dist-upgrade to 22.04: ubuntu-server depends on multipath-tools

Written by Claudio Kuenzler - 0 comments

Published on February 22nd 2023 - Listed in Linux Ubuntu Systemd

---

As the end of Ubuntu 18.04 (Bionic) is nearing, one of my tasks these days is to upgrade these machines. If possible, the machines still running Bionic would be upgraded to 22.04 (Jammy) via an intermediate upgrade to 20.04 (Focal).

During the distribution upgrade from 20.04 to 22.04 I came across an error in the upgrade process:

root@ubuntu:~# apt-get dist-upgrade
Reading package lists... Done
Building dependency tree       
Reading state information... Done
Calculating upgrade... Done
[...]
dpkg: dependency problems prevent configuration of ubuntu-server:
 ubuntu-server depends on multipath-tools; however:
  Package multipath-tools is not configured yet.

dpkg: error processing package ubuntu-server (--configure):
 dependency problems - leaving unconfigured
Setting up language-selector-common (0.219.1) ...
No apport report written because the error message indicates its a followup error from a previous failure.
[...]
Errors were encountered while processing:
 multipath-tools
 ubuntu-server
E: Sub-process /usr/bin/dpkg returned an error code (1)

At the end the apt-get dist-upgrade command existed with an error code (1). Even though the rest of the package upgrades worked fine, this particular package (ubuntu-server) threw an error because it requires multipath-tools.

Manually install multipath-tools

Let's try to manually install/upgrade the multipath-tools package:

root@ubuntu:~# apt-get install multipath-tools
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
multipath-tools is already the newest version (0.8.8-1ubuntu1.22.04.1).
[...]
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
2 not fully installed or removed.
After this operation, 0 B of additional disk space will be used.
Do you want to continue? [Y/n] y
Setting up multipath-tools (0.8.8-1ubuntu1.22.04.1) ...
Could not execute systemctl:  at /usr/bin/deb-systemd-invoke line 142.
Failed to restart multipath-tools.service: Unit multipath-tools.service failed to load properly, please adjust/correct and reload service manager: File exists
See system logs and 'systemctl status multipath-tools.service' for details.
invoke-rc.d: initscript multipath-tools, action "restart" failed.
- multipath-tools.service
     Loaded: error (Reason: Unit multipath-tools.service failed to load properly, please adjust/correct and reload service manager: File exists)
     Active: inactive (dead)
Warning: The unit file, source configuration file or drop-ins of multipath-tools.service changed on disk. Run 'systemctl daemon-reload' to reload units.
dpkg: error processing package multipath-tools (--configure):
 installed multipath-tools package post-installation script subprocess returned error exit status 1
dpkg: dependency problems prevent configuration of ubuntu-server:
 ubuntu-server depends on multipath-tools; however:
  Package multipath-tools is not configured yet.

But even this didn't work. The output shows that the package is already installed and at the currently newest version, however the service could not be started.

Service multipathd is masked

The reason for the errors during the upgrade can be seen when trying to manually enable the Systemd unit multipathd.service:

root@ubuntu:~# systemctl enable multipathd.service
Failed to enable unit: Unit file /etc/systemd/system/multipathd.service is masked.

The service was masked. This means the service cannot be started, even if forced.

Let's unmask and enable the service:

root@ubuntu:~# systemctl unmask multipathd.service
Removed /etc/systemd/system/multipathd.service.

root@ubuntu:~# systemctl enable multipathd.service

root@ubuntu:~# systemctl status multipathd.service
- multipathd.service - Device-Mapper Multipath Device Controller
     Loaded: loaded (/lib/systemd/system/multipathd.service; enabled; vendor preset: enabled)
     Active: inactive (dead)
TriggeredBy: - multipathd.socket

And now let's start multipathd:

root@ubuntu:~# systemctl start multipathd.service

root@ubuntu:~# systemctl status multipathd.service
- multipathd.service - Device-Mapper Multipath Device Controller
     Loaded: loaded (/lib/systemd/system/multipathd.service; enabled; vendor preset: enabled)
     Active: active (running) since Wed 2023-02-22 10:15:55 CET; 4s ago
TriggeredBy: - multipathd.socket
    Process: 108076 ExecStartPre=/sbin/modprobe -a scsi_dh_alua scsi_dh_emc scsi_dh_rdac dm-multipath (code=exited, s>
   Main PID: 108082 (multipathd)
     Status: "up"
      Tasks: 7
     Memory: 18.9M
     CGroup: /system.slice/multipathd.service
             |-108082 /sbin/multipathd -d -s

Continue previously broken dist-upgrade

The previous dist-upgrade can now "continue" and exited correctly this time:

root@ubuntu:~# apt-get dist-upgrade
[...]
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
1 not fully installed or removed.
After this operation, 0 B of additional disk space will be used.
Do you want to continue? [Y/n] Y
Setting up multipath-tools (0.8.8-1ubuntu1.22.04.1) ...
[...]

Alternative: Remove multipath-tools

Another alternative is to completely remove the multipath-tools package; preferrably BEFORE running apt-get dist-upgrade:

root@ubuntu:~# apt-get remove multipath-tools && apt-get purge multipath-tools

The dist-upgrade runs through just fine when multipath-tools is not installed. The error coming from the ubuntu-server package seems to only happen when multipath-tools is installed but the service is masked.

Why was multipathd.service masked?

Of course the question comes up, why multipathd.service was masked in the first place. Ubuntu does not do this by itself. The reason is that multipathd in virtual machines (mostly) doesn't make much sense. Besides this on Ubuntu 20.04 the service was VERY noisy and wrote thousands of errors into syslog:

[...]
Jun  1 10:57:23 focal multipathd[599]: sda: add missing path
Jun  1 10:57:24 focal multipathd[599]: sda: failed to get udev uid: Invalid argument
Jun  1 10:57:24 focal multipathd[599]: sda: failed to get sysfs uid: Invalid argument
Jun  1 10:57:24 focal multipathd[599]: sda: failed to get sgio uid: No such file or directory
[...]

This doesn't seem to be case in Ubuntu 22.04 anymore. The service sent the following events to syslog:

root@ubuntu:~# journalctl -u multipathd
[...]
Feb 22 10:19:20 ubuntu systemd[1]: Starting Device-Mapper Multipath Device Controller...
Feb 22 10:19:20 ubuntu multipathd[549]: --------start up--------
Feb 22 10:19:20 ubuntu multipathd[549]: read /etc/multipath.conf
Feb 22 10:19:20 ubuntu multipathd[549]: path checkers start up
Feb 22 10:19:20 ubuntu multipathd[549]: sda: failed to get udev uid: No data available
Feb 22 10:19:20 ubuntu systemd[1]: Started Device-Mapper Multipath Device Controller.
Feb 22 10:19:22 ubuntu multipathd[549]: sda: failed to get udev uid: No data available
Feb 22 10:19:22 ubuntu multipathd[549]: sda: failed to get path uid
Feb 22 10:19:22 ubuntu multipathd[549]: uevent trigger error
Feb 22 10:19:32 ubuntu multipathd[549]: sda: triggering change event to reinitialize
Feb 22 10:19:32 ubuntu multipathd[549]: sda: failed to get udev uid: No data available
Feb 22 10:19:32 ubuntu multipathd[549]: sda: failed to get path uid
Feb 22 10:19:32 ubuntu multipathd[549]: uevent trigger error
Feb 22 10:19:42 ubuntu multipathd[549]: sda: triggering change event to reinitialize
Feb 22 10:19:42 ubuntu multipathd[549]: sda: failed to get udev uid: No data available
Feb 22 10:19:42 ubuntu multipathd[549]: sda: failed to get path uid
Feb 22 10:19:42 ubuntu multipathd[549]: uevent trigger error
Feb 22 10:19:52 ubuntu multipathd[549]: sda: triggering change event to reinitialize
Feb 22 10:19:52 ubuntu multipathd[549]: sda: failed to get sysfs uid: Invalid argument
Feb 22 10:19:52 ubuntu multipathd[549]: sda: failed to get sgio uid: No such file or directory
Feb 22 10:19:52 ubuntu multipathd[549]: sda: failed to get path uid
Feb 22 10:19:52 ubuntu multipathd[549]: uevent trigger error
Feb 22 10:20:02 ubuntu multipathd[549]: sda: not initialized after 3 udev retriggers
Feb 22 10:20:03 ubuntu multipathd[549]: sda: add missing path
Feb 22 10:20:03 ubuntu multipathd[549]: sda: failed to get sysfs uid: Invalid argument
Feb 22 10:20:03 ubuntu multipathd[549]: sda: failed to get sgio uid: No such file or directory
Feb 22 10:20:03 ubuntu multipathd[549]: sda: no WWID in state "undef
Feb 22 10:20:03 ubuntu multipathd[549]: ", giving up
Feb 22 10:20:03 ubuntu multipathd[549]: sda: check_path() failed, removing

After these events, multipathd stopped trying the devices (here sda).

Add a comment

Show form to leave a comment

Comments (newest first)

No comments yet.