Today I tried to upgraded one of my monitoring machines, which also happens to run Grafana on it. Everything worked well, except the Grafana package upgrade.
root@debian:~# apt-get dist-upgrade
[...]
Preparing to unpack .../14-grafana_13.1.1_amd64.deb ...
Unpacking grafana (13.1.1) over (13.1.0) ..
[...]
Setting up grafana (13.1.1) ...
mv: cannot move '/usr/share/grafana/data/plugins-bundled' to '/var/lib/grafana/plugins-bundled': Directory not empty
dpkg: error processing package grafana (--configure):
installed grafana package post-installation script subprocess returned error exit status 1
[...]
Errors were encountered while processing:
grafana
E: Sub-process /usr/bin/dpkg returned an error code (1)
As the output shows, there was an error executing the post-installation (postinst) script.
The error is an attempted move of the directory /usr/share/grafana/data/plugins-bundled into /var/lib/grafana/ - because the destination (/var/lib/grafana/plugins-bundled) already exists and is not empty.
Once apt unpacked the deb package (which happened, as seen in the output above), you can find the postinst scripts in /var/lib/dpkg.
root@debian:~# find /var/lib/dpkg -name "*grafana*"
/var/lib/dpkg/info/grafana.list
/var/lib/dpkg/info/grafana.prerm
/var/lib/dpkg/info/grafana.conffiles
/var/lib/dpkg/info/grafana.md5sums
/var/lib/dpkg/info/grafana.postinst
Here we go, we found the post-installation script that causes the error.
Let's look for this "mv". What does it do?
root@debian:~# grep -n mv /var/lib/dpkg/info/grafana.postinst
28: mv $GRAFANA_HOME/data/plugins-bundled $DATA_DIR
So on line 28 inside the postinst script there is the infamous mv command that tries to move the plugins-bundled directory into the $DATA_DIR.
These two variables ($GRAFANA_HOME and $DATA_DIR) are defined in /etc/default/grafana-server by the way:
root@debian:~# egrep "(GRAFANA_HOME|DATA_DIR)" /etc/default/grafana-server
GRAFANA_HOME=/usr/share/grafana
DATA_DIR=/var/lib/grafana
Translated with the variables, the mv command attempts to move the /usr/share/grafana/data/plugins-bundled into /var/lib/grafana. The apt error output shows the full paths behind the variables.
Looking into the postinst file itself, something interesting shows up just in front of line 28:
27 # Move plugins-bundled into place, see https://github.com/grafana/grafana/issues/123110
28 mv $GRAFANA_HOME/data/plugins-bundled $DATA_DIR
A comment on line 27 refers to a GitHub issue. That GitHub issue attempts to solve a problem by adding this mv command into the post installation scripts in both deb and rpm packages (see pull request 125080).
This is a pretty bad idea. And most likely also goes against Debian packaging guidelines, as one of the comments mentions:
Moving some data from /usr/ to /var/ during a postinst is an absolute no-no: You mess up dpkg database which tracks which path is owned by which package
Instead of moving the full directory, I replaced line 28 in the postinst script with an rsync command, that simply synchronizes files from /usr/share/grafana/data/plugins-bundled/ to /var/lib/grafana/plugins-bundled/.
root@debian:~# sed -n '27,29p' /var/lib/dpkg/info/grafana.postinst
# Move plugins-bundled into place, see https://github.com/grafana/grafana/issues/123110
#mv $GRAFANA_HOME/data/plugins-bundled $DATA_DIR
rsync -a $GRAFANA_HOME/data/plugins-bundled/ $DATA_DIR/plugins-bundled/
The apt process now runs through fine:
root@debian:~# apt-get dist-upgrade
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
Calculating upgrade... Done
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
1 not fully installed or removed.
After this operation, 0 B of additional disk space will be used.
Do you want to continue? [Y/n] y
Setting up grafana (13.1.1) ...
Restarting grafana-server service... OK
AI AWS Android Ansible Apache Apple Atlassian BSD Backup Bash Bluecoat CMS Chef Cloud Coding Consul Containers CouchDB DB DNS Databases Docker ELK Elasticsearch Filebeat FreeBSD Galera Git GlusterFS Grafana Graphics HAProxy HTML Hacks Hardware Icinga Influx Internet Java KVM Kibana Kodi Kubernetes LVM LXC Linux Logstash Mac Macintosh Mail MariaDB Minio MongoDB Monitoring Multimedia MySQL NFS Nagios Network Nginx OSSEC OTRS Observability Office OpenSearch PHP Perl Personal PostgreSQL PowerDNS Proxmox Proxy Python Rancher Rant Redis Roundcube SSL Samba Seafile Security Shell SmartOS Solaris Surveillance Systemd TLS Tomcat Ubuntu Unix VMware Varnish Virtualization Windows Wireless Wordpress Wyse ZFS Znuny Zoneminder