Sysdig not working anymore in Docker container (empty formatting token)

Written by Claudio Kuenzler - 0 comments

Published on September 24th 2024 - Listed in Kubernetes Cloud Containers Docker Linux

---

Luckily it doesn't happen too often, but whenever I need to troubleshoot some container crazyness, I tend to use sysdig for troubleshooting. It helps to quickly identify the top cpu consumers, top network talkers, allows to run a smooth tcpdump style communication analysis and many more things really helpful for troubleshooting.

But today I ran into a problem.

empty formatting token

So first I ran the docker run command to actually download and start the sysdig container:

ck@k8snode:~# sudo docker run --rm -i -t --privileged --net=host \
    -v /var/run/docker.sock:/host/var/run/docker.sock \
    -v /dev:/host/dev \
    -v /proc:/host/proc:ro \
    -v /boot:/host/boot:ro \
    -v /src:/src \
    -v /lib/modules:/host/lib/modules:ro \
    -v /usr:/host/usr:ro \
    -v /etc:/host/etc:ro \
    docker.io/sysdig/sysdig
Unable to find image 'sysdig/sysdig:latest' locally
[...]
================ Cleaning phase ================

* Looking for a scap module locally (kernel 5.4.0-182-generic)
* Filename 'scap_ubuntu-generic_5.4.0-182-generic_202.ko' is composed of:
 - driver name: scap
 - target identifier: ubuntu-generic
 - kernel release: 5.4.0-182-generic
 - kernel version: 202
* Trying to download a prebuilt scap module from https://download.sysdig.com/scap-drivers/7.2.0%2Bdriver/x86_64/scap_ubuntu-generic_5.4.0-182-generic_202.ko
* Download succeeded
* Success: scap module found and inserted

This correctly downloaded and started the container - and at the end you're inside the container's shell. From here you can run the sysdig command:

[root@k8snode /]# sysdig -pc -c topcontainers_net
empty formatting token

Hmm... no idea why this error turns up. I referred to my own and the public documentation of sysdig and this should work.

Revert to older image

This used to work a couple of months ago, when I needed sysdig the last time. So I quickly checked the different sysdig image tags and decided to use the 0.35.0 version, which is a couple of months old.

ck@k8snode:~# sudo docker run --rm -i -t --privileged --net=host \
    -v /var/run/docker.sock:/host/var/run/docker.sock \
    -v /dev:/host/dev \
    -v /proc:/host/proc:ro \
    -v /boot:/host/boot:ro \
    -v /src:/src \
    -v /lib/modules:/host/lib/modules:ro \
    -v /usr:/host/usr:ro \
    -v /etc:/host/etc:ro \
    docker.io/sysdig/sysdig:0.35.0
Unable to find image 'sysdig/sysdig:0.35.0' locally
[...]
Building module:
cleaning build area....
'/tmp/scap-dkms-make'......
cleaning build area....

DKMS: build completed.

scap.ko:
Running module version sanity check.
 - Original module
   - No original module exists within this kernel
 - Installation
   - Installing to /lib/modules/5.4.0-182-generic/extra/
Adding any weak-modules
weak-modules: could not find dracut at /usr/bin/dracut

depmod.....

DKMS: install completed.
* scap module installed in dkms
* scap module found: /var/lib/dkms/scap/0.14.1/5.4.0-182-generic/x86_64/module/scap.ko
* Trying to insmod
* Success: scap module found and loaded in dkms

The output at the end is certainly different this time; a Kernel module (scap.ko) was built and loaded. This did not happen in the latest image.

And this time sysdig works again:

[root@k8snode /]# sysdig -pc -c topconns
Bytes               container.name      Proto               Conn                
--------------------------------------------------------------------------------
844.77KB   k8s_applicationX_ap tcp  10.42.1.225:42980->192.168.44.140:443
627.91KB   k8s_applicationY_ap tcp  10.42.3.0:51214->10.42.1.129:3005
323.76KB   k8s_applicationZ_ap tcp  127.0.0.1:37382->127.0.0.1:3000
292.94KB   k8s_applicationT_ap tcp  127.0.0.1:37232->127.0.0.1:3000
210.90KB   k8s_applicationU_ap tcp  192.168.79.18:54358->10.42.1.218:3000
210.90KB   k8s_applicationR_ap tcp  192.168.79.18:34328->10.42.1.214:3000
193.60KB   k8s_applicationA_ap tcp  10.42.7.0:45588->10.42.1.129:3005
161.72KB   k8s_applicationD_ap tcp  10.42.1.217:52666->192.168.44.140:443
147.33KB   k8s_applicationF_ap tcp  10.42.1.141:32888->192.168.44.140:443
[...]

Add a comment

Show form to leave a comment

Comments (newest first)

No comments yet.