Fatal error during Gitlab CE upgrade to 15.11.x inside a LXC container

Written by Claudio Kuenzler - 1 comments

Published on July 28th 2023 - Listed in Linux LXC Git

---

While upgrading Gitlab from an older version (14.6.1) to the newest version (currently 16.2.1), a specific upgrade path needs to be followed (more on that in another post, maybe).

During the upgrade from version 15.4.6 to 15.11.13, the upgrade process failed:

root@gitlab:~# apt-get install gitlab-ce=15.11.13-ce.0
Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following packages will be upgraded:
  gitlab-ce
[...]
---- Begin output of sysctl -e --system ----
STDOUT: * Applying /etc/sysctl.d/90-omnibus-gitlab-kernel.sem.conf ...
kernel.sem = 250 32000 32 275
* Applying /etc/sysctl.d/90-omnibus-gitlab-kernel.shmall.conf ...
kernel.shmall = 4194304
* Applying /etc/sysctl.d/90-omnibus-gitlab-kernel.shmmax.conf ...
kernel.shmmax = 17179869184
* Applying /etc/sysctl.d/90-omnibus-gitlab-net.core.somaxconn.conf ...
net.core.somaxconn = 1024
* Applying /etc/sysctl.d/99-sysctl.conf ...
* Applying /etc/sysctl.d/protect-links.conf ...
* Applying /etc/sysctl.conf ...
STDERR: sysctl: permission denied on key 'fs.protected_hardlinks'
sysctl: permission denied on key 'fs.protected_symlinks'
---- End output of sysctl -e --system ----
Ran sysctl -e --system returned 255

======
== Fatal error ==
Error updating PostgreSQL configuration. Please check the output
== Reverting ==
ok: down: postgresql: 1s, normally up
Symlink correct version of binaries: OK
ok: run: postgresql: (pid 15492) 1s
== Reverted ==
== Reverted to 12.14. Please check output for what went wrong ==
Toggling deploy page:rm -f /opt/gitlab/embedded/service/gitlab-rails/public/index.html
Toggling deploy page: OK
Toggling services:ok: run: gitaly: (pid 15504) 0s
ok: run: gitlab-kas: (pid 15531) 1s
ok: run: grafana: (pid 15548) 0s
ok: run: logrotate: (pid 15566) 0s
ok: run: sidekiq: (pid 15572) 1s
Toggling services: OK
Checking if a newer PostgreSQL version is available and attempting automatic upgrade to it: NOT OK
Error ensuring PostgreSQL is updated. Please check the logs
dpkg: error processing package gitlab-ce (--configure):
 installed gitlab-ce package post-installation script subprocess returned error exit status 1
Errors were encountered while processing:
 gitlab-ce
E: Sub-process /usr/bin/dpkg returned an error code (1)

LXC containers and Kernel parameters

Important here is to note that this Gitlab server is a LXC container. In my initial installation of this Gitlab server I already came across problems installing Gitlab in a LXC container, but they could be solved by setting the following in /etc/gitlab/gitlab.rb:

package['modify_kernel_parameters'] = false

While the other version upgrades worked fine, this time the upgrade failed - at the step of executing sysctl -e --system (shown in the output above). Let's manually check why:

root@gitlab:~# sysctl -e --system
* Applying /etc/sysctl.d/90-omnibus-gitlab-kernel.sem.conf ...
kernel.sem = 250 32000 32 275
* Applying /etc/sysctl.d/90-omnibus-gitlab-kernel.shmall.conf ...
kernel.shmall = 4194304
* Applying /etc/sysctl.d/90-omnibus-gitlab-kernel.shmmax.conf ...
kernel.shmmax = 17179869184
* Applying /etc/sysctl.d/90-omnibus-gitlab-net.core.somaxconn.conf ...
net.core.somaxconn = 1024
* Applying /etc/sysctl.d/99-sysctl.conf ...
* Applying /etc/sysctl.d/protect-links.conf ...
sysctl: permission denied on key 'fs.protected_hardlinks'
sysctl: permission denied on key 'fs.protected_symlinks'
* Applying /etc/sysctl.conf ...

The same permission denied error shows up. Let's take a look inside /etc/sysctl.d/protect-links.conf:

root@gitlab:~# cat /etc/sysctl.d/protect-links.conf
###################################################################
# Protected links
#
# Protects against creating or following links under certain conditions
# Debian kernels have both set to 1 (restricted)
# See https://www.kernel.org/doc/Documentation/sysctl/fs.txt
fs.protected_hardlinks = 1
fs.protected_symlinks = 1

OK, so this file wants to set the shown sysctl/Kernel parameters to 1 (enable). What is the current value?

root@gitlab:~# sysctl -a|grep fs.protected
fs.protected_fifos = 0
fs.protected_hardlinks = 1
fs.protected_regular = 0
fs.protected_symlinks = 1
sysctl: reading key "kernel.unprivileged_userns_apparmor_policy"

Interesting... the values are already fine. This means it's not necessary to set them (again), as this won't work anyway in a LXC container - some Kernel settings need to be done on the host, not inside the container.

Note: The same also applies to Docker containers by the way, see (Unable to deploy Kubernetes in LXC due to nf_conntrack_max).

Fixing and re-installing Gitlab

Obviously applying the fs.* Kernel settings made the Gitlab upgrade fail. Let's simply comment these out and try again:

root@gitlab:~# sed -i "s/^fs/#fs/g" /etc/sysctl.d/protect-links.conf
root@gitlab:~# sysctl -e --system
* Applying /etc/sysctl.d/90-omnibus-gitlab-kernel.sem.conf ...
kernel.sem = 250 32000 32 275
* Applying /etc/sysctl.d/90-omnibus-gitlab-kernel.shmall.conf ...
kernel.shmall = 4194304
* Applying /etc/sysctl.d/90-omnibus-gitlab-kernel.shmmax.conf ...
kernel.shmmax = 17179869184
* Applying /etc/sysctl.d/90-omnibus-gitlab-net.core.somaxconn.conf ...
net.core.somaxconn = 1024
* Applying /etc/sysctl.d/99-sysctl.conf ...
* Applying /etc/sysctl.d/protect-links.conf ...
* Applying /etc/sysctl.conf ...

No errors this time applying the remaining Kernel settings. Let's reinstall Gitlab:

root@gitlab:~# apt-get install --reinstall gitlab-ce=15.11.13-ce.0
Reading package lists... Done
Building dependency tree       
Reading state information... Done
0 upgraded, 0 newly installed, 1 reinstalled, 0 to remove and 0 not upgraded.
Need to get 0 B/1,314 MB of archives.
After this operation, 0 B of additional disk space will be used.
debconf: delaying package configuration, since apt-utils is not installed
(Reading database ... 108315 files and directories currently installed.)
Preparing to unpack .../gitlab-ce_15.11.13-ce.0_amd64.deb ...
[...]
gitlab Reconfigured!
Restarting previously running GitLab services
ok: run: gitaly: (pid 6774) 44s
ok: run: gitlab-kas: (pid 7395) 1s
ok: run: gitlab-workhorse: (pid 7411) 0s
ok: run: grafana: (pid 7428) 0s
ok: run: logrotate: (pid 7446) 1s
ok: run: nginx: (pid 7453) 0s
ok: run: postgresql: (pid 5990) 180s
ok: run: puma: (pid 7474) 1s
ok: run: redis: (pid 6262) 168s
ok: run: sidekiq: (pid 7480) 0s

     _______ __  __          __
    / ____(_) /_/ /   ____ _/ /_
   / / __/ / __/ /   / __ `/ __ \
  / /_/ / / /_/ /___/ /_/ / /_/ /
  \____/_/\__/_____/\__,_/_.___/


Upgrade complete! If your GitLab server is misbehaving try running
  sudo gitlab-ctl restart
before anything else.
If you need to roll back to the previous version you can use the database
backup made during the upgrade (scroll up for the filename).

Yay, the upgrade worked fine this time!

But what about the Gitlab setting to ignore Kernel settings?

Yeah, right. I mentioned above that my Gitlab installation in LXC worked because the Gitlab setting modify_kernel_parameters was set to false. What happened with this setting?

root@gitlab:~# cat /etc/gitlab/gitlab.rb  | grep kernel
##! Attempt to modify kernel paramaters. To skip this in containers where the
# package['modify_kernel_parameters'] = true

It turns out I never applied this setting (probably because I fixed the installation issue another way). But now is definitely the time to set this as I don't want further disruptions in the remaining upgrades:

root@gitlab:~# vi /etc/gitlab/gitlab.rb
[...]
##! Attempt to modify kernel paramaters. To skip this in containers where the
##! relevant file system is read-only, set the value to false.
# package['modify_kernel_parameters'] = true
package['modify_kernel_parameters'] = false
[...]

After the config file was adjusted, the settings can be applied using gitlab-ctl reconfigure .

Add a comment

Show form to leave a comment

Comments (newest first)

Elfy from Utah wrote on Dec 12th, 2023:

This helped me out, thank you for posting your solution!

---