Ubuntu 14.04: mailq command not working in container (Permission denied)

Written by Claudio Kuenzler - 4 comments

Published on June 8th 2015 - Listed in Linux

---

Very strange but hit me today, when I installed a new LXC host, running Ubuntu 14.04 LTS and with the latest patches. 

After having installed an LXC container and added monitoring, the check_mailq plugin didn't work anymore. A manual verification confirmed, that the mailq command could not be executed:

root@container:~# mailq
postqueue: warning: close: Permission denied

The command works on the LXC host itself though. So "only" the mailq command in a container hits that problem.

According to Ubuntu bug #1390223 and to this mailing list post, this seems to be a bug in the recently for Trusty released kernel 3.16. The original trusty kernel (3.13) didn't have this issue (I'd have seen that in previously installed containers). 

Indeed, this recently installed LXC host already includes the newer 3.16 kernel:

root@lxchost001:~# uname -a
Linux shopprodlxc001 3.16.0-30-generic #40~14.04.1-Ubuntu SMP Thu Jan 15 17:43:14 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux

Where does the kernel 3.16 come from?
This is where the big difference from Debian and Ubuntu come in. While Debian sticks with its original package versions (unless someone uses other apt sources or backports), Ubuntu often offers to install newer software packages, for example newer kernels. This has an advantage that new features can be enabled on a LTS system which was installed two years ago, but has the disadvantage of going a bit into "experimental". This is the classical example of the main difference between Debian and Ubuntu.

By checking the installed packages, one can see that the 3.16 kernel is actually the kernel version from Utopic Unicorn (aka Ubuntu 14.10):

root@lxchost001:~# dpkg -l | grep linux-image
ii  linux-image-3.16.0-30-generic       3.16.0-30.40~14.04.1   amd64   Linux kernel image for version 3.16.0 on 64 bit x86 SMP
ii  linux-image-extra-3.16.0-30-generic 3.16.0-30.40~14.04.1   amd64   Linux kernel extra modules for version 3.16.0 on 64 bit x86 SMP
ii  linux-image-generic-lts-utopic      3.16.0.30.23           amd64   Generic Linux kernel image

I am actually surprised that Ubuntu installs by default the newer available kernel and not the one intended for the LTS system. An upgrade can always easily be done with "apt-get dist-upgrade" but then the administrator knows about potential risks. If the newer kernel comes automatically on a new installed system, possible bugs can hit your system and you're in the same place as I am right now.

So let's go back to the LTS kernel 3.13 by installing the linux-image-generic-lts-trusty package (after having stopped the containers of course!):

root@lxchost001:~# apt-get install linux-image-generic-lts-trusty
Reading package lists... Done
Building dependency tree      
Reading state information... Done
The following extra packages will be installed:
  linux-image-3.13.0-53-generic linux-image-extra-3.13.0-53-generic
  linux-image-generic
Suggested packages:
  fdutils linux-doc-3.13.0 linux-source-3.13.0 linux-tools
  linux-headers-3.13.0-53-generic
The following NEW packages will be installed:
  linux-image-3.13.0-53-generic linux-image-extra-3.13.0-53-generic
  linux-image-generic linux-image-generic-lts-trusty
0 upgraded, 4 newly installed, 0 to remove and 3 not upgraded.
Need to get 51.9 MB of archives.
After this operation, 194 MB of additional disk space will be used.
[...]

When initramfs runs through as post-installation task, it finds two kernel versions:

[...]
Examining /etc/kernel/postinst.d.
run-parts: executing /etc/kernel/postinst.d/apt-auto-removal 3.13.0-53-generic /boot/vmlinuz-3.13.0-53-generic
run-parts: executing /etc/kernel/postinst.d/initramfs-tools 3.13.0-53-generic /boot/vmlinuz-3.13.0-53-generic
update-initramfs: Generating /boot/initrd.img-3.13.0-53-generic
run-parts: executing /etc/kernel/postinst.d/update-notifier 3.13.0-53-generic /boot/vmlinuz-3.13.0-53-generic
run-parts: executing /etc/kernel/postinst.d/zz-update-grub 3.13.0-53-generic /boot/vmlinuz-3.13.0-53-generic
Generating grub configuration file ...
Found linux image: /boot/vmlinuz-3.16.0-30-generic
Found initrd image: /boot/initrd.img-3.16.0-30-generic
Found linux image: /boot/vmlinuz-3.13.0-53-generic
Found initrd image: /boot/initrd.img-3.13.0-53-generic
[...]

With the current status, there would be a "dual boot" possibility with both 3.13 and 3.16 kernels. I checked /boot/grub/grub.cfg and the defailt is still set to 3.16.
In my case I only want to use 3.13, so let's remove and purge 3.16:

root@lxchost001:~# apt-get remove linux-image-generic-lts-utopic linux-image-3.16.0-30-generic linux-image-extra-3.16.0-30-generic
Reading package lists... Done
Building dependency tree      
Reading state information... Done
The following packages will be REMOVED:
  linux-generic-lts-utopic linux-image-generic-lts-utopic
0 upgraded, 0 newly installed, 2 to remove and 1 not upgraded.
After this operation, 55.3 kB disk space will be freed.
Do you want to continue? [Y/n] y
(Reading database ... 63127 files and directories currently installed.)
Removing linux-generic-lts-utopic (3.16.0.30.23) ...
Removing linux-image-generic-lts-utopic (3.16.0.30.23) ...
[...]
Removing linux-image-3.16.0-30-generic (3.16.0-30.40~14.04.1) ...
WARN: Proceeding with removing running kernel image.
Examining /etc/kernel/postrm.d .
run-parts: executing /etc/kernel/postrm.d/initramfs-tools 3.16.0-30-generic /boot/vmlinuz-3.16.0-30-generic
update-initramfs: Deleting /boot/initrd.img-3.16.0-30-generic
run-parts: executing /etc/kernel/postrm.d/zz-update-grub 3.16.0-30-generic /boot/vmlinuz-3.16.0-30-generic
Generating grub configuration file ...
Found linux image: /boot/vmlinuz-3.13.0-53-generic
Found initrd image: /boot/initrd.img-3.13.0-53-generic
Found memtest86+ image: /boot/memtest86+.elf
Found memtest86+ image: /boot/memtest86+.bin
done
The link /vmlinuz.old is a damaged link
Removing symbolic link vmlinuz.old
 you may need to re-run your boot loader[grub]
The link /initrd.img.old is a damaged link
Removing symbolic link initrd.img.old
 you may need to re-run your boot loader[grub]

The warnings are pretty good, so to avoid trying to boot a non-existant kernel, we update the grub configs:

root@lxchost001:~# update-grub
Generating grub configuration file ...
Found linux image: /boot/vmlinuz-3.13.0-53-generic
Found initrd image: /boot/initrd.img-3.13.0-53-generic
Found memtest86+ image: /boot/memtest86+.elf
Found memtest86+ image: /boot/memtest86+.bin
done

The system now needs to be booted with the 3.13 trusty kernel, so reboot:

root@lxchost001:~# reboot

Once the system is back up again, verify what kernel version is now being used:

root@lxchost001:~# uname -a
Linux shopprodlxc002 3.13.0-53-generic #89-Ubuntu SMP Wed May 20 10:34:39 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux

root@lxchost001:~# dpkg -l | grep linux-image
ii  linux-image-3.13.0-53-generic       3.13.0-53.89                     amd64        Linux kernel image for version 3.13.0 on 64 bit x86 SMP
rc  linux-image-3.16.0-30-generic       3.16.0-30.40~14.04.1             amd64        Linux kernel image for version 3.16.0 on 64 bit x86 SMP
ii  linux-image-extra-3.13.0-53-generic 3.13.0-53.89                     amd64        Linux kernel extra modules for version 3.13.0 on 64 bit x86 SMP
rc  linux-image-extra-3.16.0-30-generic 3.16.0-30.40~14.04.1             amd64        Linux kernel extra modules for version 3.16.0 on 64 bit x86 SMP
ii  linux-image-generic                 3.13.0.53.60                     amd64        Generic Linux kernel image
ii  linux-image-generic-lts-trusty      3.13.0.53.60                     amd64        Generic Linux kernel image

The 3.16 packages are still on the file system, but are not installed anymore. They can now be purged. But important is, that the 3.13 kernel is now active.

Let's check if the mailq command now works again in the container:

root@container:~# mailq
Mail queue is empty

Yes, it's working again!

And this is why my personal preference is to use Debian systems for stability. But to use the new LXC features Ubuntu is currently the better choice. Like in life, it's always about compromise.

Add a comment

Show form to leave a comment

Comments (newest first)

ck from Switzerland wrote on Aug 18th, 2016:

thats probably because you have the package linux-generic-lts-utopic installed. According to the changelog (http://changelogs.ubuntu.com/changelogs/pool/main/l/linux-meta-lts-utopic/linux-meta-lts-utopic_3.16.0.77.68/changelog) you should be able to update to 3.16.0.77.68. If you want a newer kernel (for example 3.19) you can install the linux-generic-lpae-lts-vivid meta package.

---

Andrey from wrote on Aug 18th, 2016:

Hi. I have 3.16.0-50 and have no updates for it. I don't know why.

---

ck from Switzerland wrote on Aug 18th, 2016:

Hi Andrey, meanwhile I have the newer 3.16 kernel on the LXC hosts and I havent seen any problems anymore. Current kernel version on my LXC hosts is 3.16.0-50-generic.

---

Andrey from wrote on Aug 18th, 2016:

Thanks a lot for solution. But what can you tell about makeing updates on this server? If I further try "apt-get upgrade" will I obtain same 3.16? It will be unupdated server?

---