How to fix broken apt dist-upgrade when /var is out of disk space

Written by Claudio Kuenzler - 0 comments

Published on September 9th 2022 - Listed in Linux

---

When running apt-get dist-upgrade (or the newer apt upgrade), the command downloads the update packages (.deb) into /var/cache/apt/archives. Once downloaded, apt runs through the package installers. Depending on each package, there might be additional pre- or post-install steps to be done.

One of that is the upgrade of a linux-image (Kernel) or a Kernel related package (e.g. linux-firmware). It launches update-initramfs in the background, updating the initramfs for the new Kernel version. This process uses /var/tmp to build the new initramfs.

Diskspace (/var) full during upgrade

If you happen to be in the unlucky situation that your /var partition (or root partition if you don't have a separate partition for /var) is filled before the apt process is completed, you run into a dangerous situation:

root@ubuntu:~# apt-get update && apt-get dist-upgrade
[...]
Setting up linux-firmware (1.187.33) ...
update-initramfs: Generating /boot/initrd.img-5.4.0-97-generic
depmod: ERROR: Could not create index 'modules.dep'. Output is truncated: No space left on device
cp: error writing '/var/tmp/mkinitramfs_ELDs7f/etc/ld.so.conf': No space left on device
cp: cannot create directory '/var/tmp/mkinitramfs_ELDs7f/etc/ld.so.conf.d': No space left on device
/sbin/ldconfig.real: Writing of cache data failed: No space left on device
cp: error writing '/var/tmp/mkinitramfs_ELDs7f//usr/lib/x86_64-linux-gnu/libnss_dns-2.31.so': No space left on device
cp: error writing '/var/tmp/mkinitramfs_ELDs7f//usr/lib/x86_64-linux-gnu/libresolv-2.31.so': No space left on device
update-initramfs: Generating /boot/initrd.img-5.4.0-96-generic
cp: error writing '/var/tmp/mkinitramfs_cAbtxh//usr/lib/x86_64-linux-gnu/libkmod.so.2.3.5': No space left on device
E: /usr/share/initramfs-tools/hooks/udev failed with return 1.
update-initramfs: failed for /boot/initrd.img-5.4.0-96-generic with 1.
dpkg: error processing package linux-firmware (--configure):
 installed linux-firmware package post-installation script subprocess returned error exit status 1
[...]
Processing triggers for initramfs-tools (0.136ubuntu6.7) ...
update-initramfs: Generating /boot/initrd.img-5.4.0-125-generic
Processing triggers for systemd (245.4-4ubuntu3.17) ...
Processing triggers for dbus (1.12.16-2ubuntu2.2) ...
Errors were encountered while processing:
 linux-firmware
 linux-image-generic
 linux-generic
E: Sub-process /usr/bin/dpkg returned an error code (1)

A reboot of the server in this situation means trouble: The Ubuntu server will most likely not be able to start anymore.

Make or increase available space

Luckily there's no need to panic, the errors above can be fixed. But first you need to find out why /var is full. This could be because of a lot of logs in /var/log, it could be huge temp files in /var/tmp, it could be anything. In this particular situation the cause was a Logstash process logging a lot of errors into /var/log/logstash, filling up the /var partition very quickly. Deleting the Logstash (plain) logs gave enough available disk space back.

Another alternative is of course to increase the /var partition. That's easy when you are using Logical Volumes - it might be trickier if you have /var on your root partition.

Once you got some space back on /var, you can continue with fixing apt.

Fixing the previously broken upgrade

apt, respectively dpkg, know which packages were broken during the upgrade and will attempt to launch the failed process (e.g. postinstall scripts) again.

This can be initiated with dpkg --configure -a:

root@ubuntu:~# dpkg --configure -a
Setting up linux-firmware (1.187.33) ...
update-initramfs: Generating /boot/initrd.img-5.4.0-125-generic
update-initramfs: Generating /boot/initrd.img-5.4.0-97-generic
update-initramfs: Generating /boot/initrd.img-5.4.0-96-generic
update-initramfs: Generating /boot/initrd.img-5.4.0-94-generic
update-initramfs: Generating /boot/initrd.img-5.4.0-42-generic
Setting up linux-image-generic (5.4.0.125.126) ...
Setting up linux-generic (5.4.0.125.126) ...

The initramfs was successfully built (this time). With another apt-get dist-upgrade you can verify that there is indeed no problem left:

root@ubuntu:~# apt-get dist-upgrade
Reading package lists... Done
Building dependency tree       
Reading state information... Done
Calculating upgrade... Done
The following packages were automatically installed and are no longer required:
  linux-headers-5.4.0-94 linux-headers-5.4.0-94-generic linux-headers-5.4.0-96 linux-headers-5.4.0-96-generic linux-headers-5.4.0-97 linux-headers-5.4.0-97-generic linux-image-5.4.0-94-generic linux-image-5.4.0-96-generic
  linux-image-5.4.0-97-generic linux-modules-5.4.0-94-generic linux-modules-5.4.0-96-generic linux-modules-5.4.0-97-generic linux-modules-extra-5.4.0-94-generic linux-modules-extra-5.4.0-96-generic linux-modules-extra-5.4.0-97-generic
Use 'apt autoremove' to remove them.
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.

Add a comment

Show form to leave a comment

Comments (newest first)

No comments yet.