Icinga 2 with MySQL IDO fails to start after Debian release (distribution) upgrade

The same also goes for the Icinga 2 package repositories: The repos were changed from icinga-stretch to icinga-buster.

However after Debian was upgraded, Icinga failed to (re-) start.

root@icinga2:/# systemctl status icinga2
-  icinga2.service - Icinga host/service/network monitoring system
   Loaded: loaded (/lib/systemd/system/icinga2.service; enabled; vendor preset: enabled)
  Drop-In: /etc/systemd/system/icinga2.service.d
           |-00-restart.conf, limits.conf
   Active: failed (Result: exit-code) since Sat 2021-05-01 16:31:32 CEST; 2min 49s ago
  Process: 1266 ExecStartPre=/usr/lib/icinga2/prepare-dirs /etc/default/icinga2 (code=exited, status=0/SUCCESS)
  Process: 1271 ExecStart=/usr/sbin/icinga2 daemon --close-stdio -e ${ICINGA2_ERROR_LOG} (code=exited, status=1/FAILURE)
 Main PID: 1271 (code=exited, status=1/FAILURE)

May 01 16:31:32 icinga2 systemd[1]: icinga2.service: Service RestartSec=100ms expired, scheduling restart.
May 01 16:31:32 icinga2 systemd[1]: icinga2.service: Scheduled restart job, restart counter is at 5.
May 01 16:31:32 icinga2 systemd[1]: Stopped Icinga host/service/network monitoring system.
May 01 16:31:32 icinga2 systemd[1]: icinga2.service: Start request repeated too quickly.
May 01 16:31:32 icinga2 systemd[1]: icinga2.service: Failed with result 'exit-code'.
May 01 16:31:32 icinga2 systemd[1]: Failed to start Icinga host/service/network monitoring system. 

Analysis: Icinga 2 Packages not upgraded

The Systemd output above was not very helpful in finding out the reason of Icinga 2 not restarting. However doing an Icinga 2 configuration check revealed much more:

root@icinga2:/# /etc/init.d/icinga2 checkconfig
[....] checking Icinga2 configuration
[2021-05-01 16:34:31 +0200] information/cli: Icinga application loader (version: r2.12.3-1)
[2021-05-01 16:34:31 +0200] information/cli: Loading configuration file(s).
[2021-05-01 16:34:31 +0200] information/ConfigItem: Committing config item(s).
[2021-05-01 16:34:31 +0200] critical/config: Error: Could not load library 'libmysql_shim.so.2.12.3': libmysql_shim.so.2.12.3: cannot open shared object file: No such file or directory

[2021-05-01 16:34:31 +0200] critical/config: 1 error
[2021-05-01 16:34:31 +0200] critical/cli: Config validation failed. Re-run with 'icinga2 daemon -C' after fixing the config.
[FAIL] checking Icinga2 configuration. Check '/var/log/icinga2/startup.log' for details. ... failed!

Note: Using a pre-Systemd service unit init script's checkconfig option is the same as running icinga2 daemon -C.

Obivously Icinga 2 can't find a required library related to MySQL. This error somehow related to a problem I wrote about in August 2020 (Upgrading Icinga 2 from 2.10 to 2.12 on Debian Stretch fails due to missing backports), but this time the Debian backports repositories are enabled. However the reason for the error is the same: The icinga2-ido-mysql package was not installed anymore (rc means removed) after the Debian release upgrade:

root@icinga2:/# dpkg -l|grep icinga2
ii  icinga2-bin                          2.12.3-1.stretch              amd64        host and network monitoring system - daemon
ii  icinga2-common                       2.12.3-1.stretch              all          host and network monitoring system - common files
ii  icinga2-doc                          2.12.3-1.stretch              all          host and network monitoring system - documentation
rc  icinga2-ido-mysql                    2.12.3-1.stretch              amd64        host and network monitoring system - MySQL support

Even more intriguing: The icinga2 packages still remained at the stretch version. The apt history logs reveal that no icinga2 packages were upgraded during the apt-get dist-upgrade process - on the other hand the package icinga2-ido-mysql was removed:

root@icinga2:/var/log/apt# cat history.log
Start-Date: 2021-05-01  16:17:22
Commandline: apt-get dist-upgrade
Install: [lots of packages]
Upgrade: [lots of packages - but no icinga2 packages]
Remove: [...] icinga2-ido-mysql:amd64 (2.12.3-1.stretch), [...]
End-Date: 2021-05-01  16:20:59

An attempt to manually install the removed package now results in a dependency/repository conflict:

root@icinga2:/# apt-get install icinga2-ido-mysql
Reading package lists... Done
Building dependency tree       
Reading state information... Done
Some packages could not be installed. This may mean that you have
requested an impossible situation or if you are using the unstable
distribution that some required packages have not yet been created
or been moved out of Incoming.
The following information may help to resolve the situation:

The following packages have unmet dependencies:
 icinga2-ido-mysql : Depends: icinga2-bin (= 2.12.3-1.buster) but 2.12.3-1.stretch is to be installed
                     Depends: icinga2-common (= 2.12.3-1.buster) but 2.12.3-1.stretch is to be installed
E: Unable to correct problems, you have held broken packages.

The reason

So why did this actually happen? Let's take a closer look at the names of the icinga2 Debian packages:

icinga2-bin:amd64 (2.12.3-1.stretch)
icinga2-bin:amd64 (2.12.3-1.buster)

icinga2-doc:amd64 (2.12.3-1.stretch)
icinga2-doc:amd64 (2.12.3-1.buster)

icinga2-common:amd64 (2.12.3-1.stretch)
icinga2-common:amd64 (2.12.3-1.buster)

icinga2-ido-mysql:amd64 (2.12.3-1.stretch)
icinga2-ido-mysql:amd64 (2.12.3-1.buster)

Note the identical version numbers? The only difference is the Debian release (code-) name: stretch and buster. The APT package uses numerical and alphabetical sorting to find out whether a new version of the package is available. In this case the ordering happens alphabetically - meaning S (stretch) comes after B (buster) and is therefore newer. Now it makes sense, why apt-get dist-upgrade did not upgrade any icinga2 packages.

To solve this, the Debian packages need to be renamed, for example using the Debian release number (not name). This way each Debian release increases in number and APT is not confused anymore. An issue was opened in the Icinga 2 code repositories on GitHub.

Note: The same problem also applies to Ubuntu packages.

Workarounds

Of course there are workarounds :-). 

Workaround 1: Remove packages and re-install

By removing the icinga2 packages and then installing them again will pick them up from the correct (buster) repository and the dependency conflicts are gone. Luckily a removal of the package does not delete your config files in /etc/icinga2, so this is pretty safe to do:

root@icinga2:~# apt-get remove icinga2-bin icinga2-common icinga2-doc

root@icinga2:~# dpkg -l|grep icinga2
rc  icinga2-bin                          2.12.3-1.stretch              amd64        host and network monitoring system - daemon
rc  icinga2-common                       2.12.3-1.stretch              all          host and network monitoring system - common files
rc  icinga2-ido-mysql                    2.12.3-1.stretch              amd64        host and network monitoring system - MySQL support

root@icinga2:~# apt-get install icinga2-bin icinga2-common icinga2-doc icinga2-ido-mysql

root@icinga2:~# dpkg -l|grep icinga2
ii  icinga2-bin                          2.12.3-1.buster               amd64        host and network monitoring system - daemon
ii  icinga2-common                       2.12.3-1.buster               all          host and network monitoring system - common files
ii  icinga2-doc                          2.12.3-1.buster               all          host and network monitoring system - documentation
ii  icinga2-ido-mysql                    2.12.3-1.buster               amd64        host and network monitoring system - MySQL support

An Icinga 2 restart is now finally working again:

root@icinga2:~# systemctl restart icinga2
root@icinga2:~# systemctl status icinga2
- icinga2.service - Icinga host/service/network monitoring system
   Loaded: loaded (/lib/systemd/system/icinga2.service; enabled; vendor preset: enabled)
  Drop-In: /etc/systemd/system/icinga2.service.d
           |-00-restart.conf, limits.conf
   Active: active (running) since Sat 2021-05-01 16:45:36 CEST; 7s ago
  Process: 4603 ExecStartPre=/usr/lib/icinga2/prepare-dirs /etc/default/icinga2 (code=exited, status=0/SUCCESS)
 Main PID: 4608 (icinga2)
    Tasks: 126
   Memory: 62.4M
   CGroup: /system.slice/icinga2.service

Workaround 2: Launch a package downgrade

By manually forcing APT to use the buster version of the packages (2.12.3-1.buster with today's version), the package manager will accept to DOWNGRADE the packages (again: downgrade because of the "older name"):

root@icinga2:~# apt-get install icinga2=2.12.3-1.buster icinga2-bin=2.12.3-1.buster icinga2-common=2.12.3-1.buster icinga2-doc=2.12.3-1.buster icinga2-ido-mysql=2.12.3-1.buster
Reading package lists... Done
Building dependency tree       
Reading state information... Done
Suggested packages:
  vim-icinga2 default-mysql-server | mysql-server
Recommended packages:
  icinga2-doc
The following NEW packages will be installed:
  icinga2-ido-mysql
The following packages will be DOWNGRADED:
  icinga2 icinga2-bin icinga2-common icinga2-doc
0 upgraded, 1 newly installed, 4 downgraded, 0 to remove and 0 not upgraded.
Need to get 10.1 MB of archives.
After this operation, 508 kB of additional disk space will be used.
Do you want to continue? [Y/n] y

This also results in the buster packages being installed, and Icinga 2 restart works again.

root@icinga2:~# dpkg -l|grep icinga2
ii  icinga2                              2.12.3-1.buster              amd64        host and network monitoring system
ii  icinga2-bin                          2.12.3-1.buster              amd64        host and network monitoring system - daemon
ii  icinga2-common                       2.12.3-1.buster              all          host and network monitoring system - common files
ii  icinga2-doc                          2.12.3-1.buster              all          host and network monitoring system - documentation
ii  icinga2-ido-mysql                    2.12.3-1.buster              amd64        host and network monitoring system - MySQL support

Anti Spam Question: What's the short name for Wireless LAN? All capital letters.