Kubernetes cluster (initiated with kubeadm) not starting after a reboot

Written by Claudio Kuenzler - 0 comments

Published on November 2nd 2022 - last updated on February 15th 2023 - Listed in Kubernetes Linux Containers

---

In a lab environment I was working on a Kubernetes cluster (v1.25.1) which was started (initiated) using kubeadm. The Kubernetes cluster consisted of a single control plane (cp) node and a worker node. However after a reboot of the lab environment the cluster didn't come up anymore.

This blog post is about issues with kubelet related to enabled swap. Another possible reason could be missing Kernel modules (such as br_netfilter). Check out the follow-up article to find out how to do automatic modprobe (enable Linux Kernel modules) at boot time.

The kubelet service should be responsible to start up the cluster automatically after a reboot yet kubectl was unable to connect to the Kubernetes API server.

On the worker node the same issue happened, the kubelet service failed to start. Checking the logs revealed something interesting:

root@cka2:~# systemctl status kubelet
- kubelet.service - kubelet: The Kubernetes Node Agent
     Loaded: loaded (/lib/systemd/system/kubelet.service; enabled; vendor preset: enabled)
    Drop-In: /etc/systemd/system/kubelet.service.d
             |-10-kubeadm.conf
     Active: activating (auto-restart) (Result: exit-code) since Wed 2022-11-02 07:45:13 CET; 3s ago
       Docs: https://kubernetes.io/docs/home/
    Process: 3324 ExecStart=/usr/bin/kubelet $KUBELET_KUBECONFIG_ARGS $KUBELET_CONFIG_ARGS $KUBELET_KUBEADM_ARGS $KUBELET_EXTRA_ARGS (code=exited, status=1/FAILURE)
   Main PID: 3324 (code=exited, status=1/FAILURE)

root@cka2:~# journalctl -fu kubelet
-- Logs begin at Mon 2022-06-20 16:00:33 CEST. --
Nov 02 07:45:34 cka2 kubelet[3377]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.27. Image garbage collector will get sandbox image information from CRI.
Nov 02 07:45:34 cka2 kubelet[3377]: I1102 07:45:34.229767    3377 server.go:413] "Kubelet version" kubeletVersion="v1.25.1"
Nov 02 07:45:34 cka2 kubelet[3377]: I1102 07:45:34.229829    3377 server.go:415] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Nov 02 07:45:34 cka2 kubelet[3377]: I1102 07:45:34.229998    3377 server.go:825] "Client rotation is on, will bootstrap in background"
Nov 02 07:45:34 cka2 kubelet[3377]: I1102 07:45:34.231249    3377 certificate_store.go:130] Loading cert/key pair from "/var/lib/kubelet/pki/kubelet-client-current.pem".
Nov 02 07:45:34 cka2 kubelet[3377]: I1102 07:45:34.232066    3377 dynamic_cafile_content.go:157] "Starting controller" name="client-ca-bundle::/etc/kubernetes/pki/ca.crt"
Nov 02 07:45:34 cka2 kubelet[3377]: I1102 07:45:34.236688    3377 server.go:660] "--cgroups-per-qos enabled, but --cgroup-root was not specified.  defaulting to /"
Nov 02 07:45:34 cka2 kubelet[3377]: E1102 07:45:34.236945    3377 run.go:74] "command failed" err="failed to run Kubelet: running with swap on is not supported, please disable swap! or set --fail-swap-on flag to false. /proc/swaps contained: [Filename\t\t\t\tType\t\tSize\tUsed\tPriority /swapfile                               file\t\t1190340\t0\t-2]"
Nov 02 07:45:34 cka2 systemd[1]: kubelet.service: Main process exited, code=exited, status=1/FAILURE
Nov 02 07:45:34 cka2 systemd[1]: kubelet.service: Failed with result 'exit-code'.

The reason might sound stupid but it's so easy to fix: swap needs to be disabled!

To do this, first correctly stop the kubelet service:

root@cka2:~# systemctl stop kubelet

Disable swap on all Kubernetes nodes:

root@cka2:~# swapoff -a

And then start the kubelet service (first on the cp node) again:

 root@cka2:~# systemctl start kubelet

The kubelet service is now successfully started and also the Kubernetes cluster is up again:

root@cka2:~# systemctl status kubelet
- kubelet.service - kubelet: The Kubernetes Node Agent
     Loaded: loaded (/lib/systemd/system/kubelet.service; enabled; vendor preset: enabled)
    Drop-In: /etc/systemd/system/kubelet.service.d
             |-10-kubeadm.conf
     Active: active (running) since Wed 2022-11-02 07:45:59 CET; 14s ago
       Docs: https://kubernetes.io/docs/home/
   Main PID: 3446 (kubelet)
      Tasks: 16 (limit: 9416)
[...]

ckadm@cka1:~$ kubectl get node
NAME   STATUS   ROLES           AGE     VERSION
cka1   Ready    control-plane   5d18h   v1.25.1
cka2   Ready    <none>          5d17h   v1.25.1

Add a comment

Show form to leave a comment

Comments (newest first)

No comments yet.