LXC container unable to start: Failed to setup cgroup limits for container

Written by Claudio Kuenzler - 0 comments

Published on March 9th 2023 - Listed in LXC Linux

---

A newly created LXC container with cgroup limits (e.g. to set a capacity of memory or vcpus) was unable to start but did not show why in the output.

In such a situation it's always a good idea to try start the container with DEBUG logs (-l DEBUG) and write the container output to a file (-o file).

root@host ~ # lxc-start -n donkey -o lxc.log -l DEBUG -L console.log
lxc-start: donkey: lxccontainer.c: wait_on_daemonized_start: 842 Received container state "ABORTING" instead of "RUNNING"
lxc-start: donkey: tools/lxc_start.c: main: 330 The container failed to start
lxc-start: donkey: tools/lxc_start.c: main: 333 To get more details, run the container in foreground mode
lxc-start: donkey: tools/lxc_start.c: main: 336 Additional information can be obtained by setting the --logfile and --logpriority options

Unless an absolute path was given to the -o option, the output file is now created in the same path where you launched the command, so you can read the file from the same path:

root@host ~ # cat lxc.log
lxc-start donkey 20230308172309.536 INFO     lxccontainer - lxccontainer.c:do_lxcapi_start:961 - Set process title to [lxc monitor] /var/lib/lxc donkey
lxc-start donkey 20230308172309.542 INFO     lsm - lsm/lsm.c:lsm_init:50 - LSM security driver AppArmor
lxc-start donkey 20230308172309.544 INFO     seccomp - seccomp.c:parse_config_v2:759 - Processing "reject_force_umount  # comment this to allow umount -f;  not recommended"
lxc-start donkey 20230308172309.544 INFO     seccomp - seccomp.c:do_resolve_add_rule:505 - Set seccomp rule to reject force umounts
lxc-start donkey 20230308172309.544 INFO     seccomp - seccomp.c:parse_config_v2:937 - Added native rule for arch 0 for reject_force_umount action 0(kill)
lxc-start donkey 20230308172309.544 INFO     seccomp - seccomp.c:do_resolve_add_rule:505 - Set seccomp rule to reject force umounts
lxc-start donkey 20230308172309.544 INFO     seccomp - seccomp.c:parse_config_v2:946 - Added compat rule for arch 1073741827 for reject_force_umount action 0(kill)
lxc-start donkey 20230308172309.544 INFO     seccomp - seccomp.c:do_resolve_add_rule:505 - Set seccomp rule to reject force umounts
lxc-start donkey 20230308172309.544 INFO     seccomp - seccomp.c:parse_config_v2:956 - Added compat rule for arch 1073741886 for reject_force_umount action 0(kill)
lxc-start donkey 20230308172309.545 INFO     seccomp - seccomp.c:do_resolve_add_rule:505 - Set seccomp rule to reject force umounts
lxc-start donkey 20230308172309.545 INFO     seccomp - seccomp.c:parse_config_v2:966 - Added native rule for arch -1073741762 for reject_force_umount action 0(kill)
lxc-start donkey 20230308172309.545 INFO     seccomp - seccomp.c:parse_config_v2:759 - Processing "[all]"
lxc-start donkey 20230308172309.545 INFO     seccomp - seccomp.c:parse_config_v2:759 - Processing "kexec_load errno 1"
lxc-start donkey 20230308172309.545 INFO     seccomp - seccomp.c:parse_config_v2:937 - Added native rule for arch 0 for kexec_load action 327681(errno)
lxc-start donkey 20230308172309.545 INFO     seccomp - seccomp.c:parse_config_v2:946 - Added compat rule for arch 1073741827 for kexec_load action 327681(errno)
lxc-start donkey 20230308172309.545 INFO     seccomp - seccomp.c:parse_config_v2:956 - Added compat rule for arch 1073741886 for kexec_load action 327681(errno)
lxc-start donkey 20230308172309.545 INFO     seccomp - seccomp.c:parse_config_v2:966 - Added native rule for arch -1073741762 for kexec_load action 327681(errno)
lxc-start donkey 20230308172309.545 INFO     seccomp - seccomp.c:parse_config_v2:759 - Processing "open_by_handle_at errno 1"
lxc-start donkey 20230308172309.545 INFO     seccomp - seccomp.c:parse_config_v2:937 - Added native rule for arch 0 for open_by_handle_at action 327681(errno)
lxc-start donkey 20230308172309.546 INFO     seccomp - seccomp.c:parse_config_v2:946 - Added compat rule for arch 1073741827 for open_by_handle_at action 327681(errno)
lxc-start donkey 20230308172309.546 INFO     seccomp - seccomp.c:parse_config_v2:956 - Added compat rule for arch 1073741886 for open_by_handle_at action 327681(errno)
lxc-start donkey 20230308172309.546 INFO     seccomp - seccomp.c:parse_config_v2:966 - Added native rule for arch -1073741762 for open_by_handle_at action 327681(errno)
lxc-start donkey 20230308172309.546 INFO     seccomp - seccomp.c:parse_config_v2:759 - Processing "init_module errno 1"
lxc-start donkey 20230308172309.546 INFO     seccomp - seccomp.c:parse_config_v2:937 - Added native rule for arch 0 for init_module action 327681(errno)
lxc-start donkey 20230308172309.546 INFO     seccomp - seccomp.c:parse_config_v2:946 - Added compat rule for arch 1073741827 for init_module action 327681(errno)
lxc-start donkey 20230308172309.546 INFO     seccomp - seccomp.c:parse_config_v2:956 - Added compat rule for arch 1073741886 for init_module action 327681(errno)
lxc-start donkey 20230308172309.546 INFO     seccomp - seccomp.c:parse_config_v2:966 - Added native rule for arch -1073741762 for init_module action 327681(errno)
lxc-start donkey 20230308172309.547 INFO     seccomp - seccomp.c:parse_config_v2:759 - Processing "finit_module errno 1"
lxc-start donkey 20230308172309.547 INFO     seccomp - seccomp.c:parse_config_v2:937 - Added native rule for arch 0 for finit_module action 327681(errno)
lxc-start donkey 20230308172309.547 INFO     seccomp - seccomp.c:parse_config_v2:946 - Added compat rule for arch 1073741827 for finit_module action 327681(errno)
lxc-start donkey 20230308172309.547 INFO     seccomp - seccomp.c:parse_config_v2:956 - Added compat rule for arch 1073741886 for finit_module action 327681(errno)
lxc-start donkey 20230308172309.547 INFO     seccomp - seccomp.c:parse_config_v2:966 - Added native rule for arch -1073741762 for finit_module action 327681(errno)
lxc-start donkey 20230308172309.547 INFO     seccomp - seccomp.c:parse_config_v2:759 - Processing "delete_module errno 1"
lxc-start donkey 20230308172309.547 INFO     seccomp - seccomp.c:parse_config_v2:937 - Added native rule for arch 0 for delete_module action 327681(errno)
lxc-start donkey 20230308172309.547 INFO     seccomp - seccomp.c:parse_config_v2:946 - Added compat rule for arch 1073741827 for delete_module action 327681(errno)
lxc-start donkey 20230308172309.547 INFO     seccomp - seccomp.c:parse_config_v2:956 - Added compat rule for arch 1073741886 for delete_module action 327681(errno)
lxc-start donkey 20230308172309.548 INFO     seccomp - seccomp.c:parse_config_v2:966 - Added native rule for arch -1073741762 for delete_module action 327681(errno)
lxc-start donkey 20230308172309.548 INFO     seccomp - seccomp.c:parse_config_v2:970 - Merging compat seccomp contexts into main context
lxc-start donkey 20230308172309.555 DEBUG    terminal - terminal.c:lxc_terminal_peer_default:707 - No such device - The process does not have a controlling terminal
lxc-start donkey 20230308172309.555 DEBUG    terminal - terminal.c:lxc_terminal_create_log_file:879 - Using "console.log" as terminal log file
lxc-start donkey 20230308172309.222 INFO     start - start.c:lxc_init:904 - Container "donkey" is initialized
lxc-start donkey 20230308172309.224 INFO     network - network.c:instantiate_veth:147 - Retrieved mtu 1500 from virbr1
lxc-start donkey 20230308172309.224 INFO     network - network.c:instantiate_veth:175 - Attached "veth0-axe26" to bridge "virbr1"
lxc-start donkey 20230308172309.225 DEBUG    network - network.c:instantiate_veth:201 - Instantiated veth "veth0-axe26/vethJU163B", index is "96"
lxc-start donkey 20230308172309.225 DEBUG    cgfsng - cgroups/cgfsng.c:cg_legacy_handle_cpuset_hierarchy:620 - "cgroup.clone_children" was already set to "1"
lxc-start donkey 20230308172309.228 INFO     start - start.c:lxc_spawn:1700 - Cloned CLONE_NEWNS
lxc-start donkey 20230308172309.228 INFO     start - start.c:lxc_spawn:1700 - Cloned CLONE_NEWPID
lxc-start donkey 20230308172309.228 INFO     start - start.c:lxc_spawn:1700 - Cloned CLONE_NEWUTS
lxc-start donkey 20230308172309.228 INFO     start - start.c:lxc_spawn:1700 - Cloned CLONE_NEWIPC
lxc-start donkey 20230308172309.228 INFO     start - start.c:lxc_spawn:1700 - Cloned CLONE_NEWNET
lxc-start donkey 20230308172309.228 DEBUG    start - start.c:lxc_try_preserve_namespaces:196 - Preserved mnt namespace via fd 15
lxc-start donkey 20230308172309.228 DEBUG    start - start.c:lxc_try_preserve_namespaces:196 - Preserved pid namespace via fd 16
lxc-start donkey 20230308172309.228 DEBUG    start - start.c:lxc_try_preserve_namespaces:196 - Preserved uts namespace via fd 17
lxc-start donkey 20230308172309.228 DEBUG    start - start.c:lxc_try_preserve_namespaces:196 - Preserved ipc namespace via fd 18
lxc-start donkey 20230308172309.228 DEBUG    start - start.c:lxc_try_preserve_namespaces:196 - Preserved net namespace via fd 19
lxc-start donkey 20230308172309.228 WARN     cgfsng - cgroups/cgfsng.c:__cg_legacy_setup_limits:2228 - Failed to set "cpuset.cpus" to "23-24"
lxc-start donkey 20230308172309.228 ERROR    start - start.c:lxc_spawn:1735 - Failed to setup cgroup limits for container "donkey"
lxc-start donkey 20230308172309.294 INFO     network - network.c:lxc_delete_network_priv:2594 - Removed interface "(null)" with index 96
lxc-start donkey 20230308172309.299 WARN     network - network.c:lxc_delete_network_priv:2613 - Invalid argument - Failed to remove interface "veth0-axe26" from "virbr1"
lxc-start donkey 20230308172309.299 DEBUG    network - network.c:lxc_delete_network:3180 - Deleted network devices
lxc-start donkey 20230308172309.299 DEBUG    lxccontainer - lxccontainer.c:wait_on_daemonized_start:830 - First child 22114 exited
lxc-start donkey 20230308172309.299 ERROR    lxccontainer - lxccontainer.c:wait_on_daemonized_start:842 - Received container state "ABORTING" instead of "RUNNING"
lxc-start donkey 20230308172309.299 ERROR    lxc_start - tools/lxc_start.c:main:330 - The container failed to start
lxc-start donkey 20230308172309.299 ERROR    lxc_start - tools/lxc_start.c:main:333 - To get more details, run the container in foreground mode
lxc-start donkey 20230308172309.299 ERROR    lxc_start - tools/lxc_start.c:main:336 - Additional information can be obtained by setting the --logfile and --logpriority options
lxc-start donkey 20230308172309.299 ERROR    start - start.c:__lxc_start:1951 - Failed to spawn container "donkey"
lxc-start donkey 20230308172309.480 INFO     conf - conf.c:run_script_argv:356 - Executing script "/usr/share/lxcfs/lxc.reboot.hook" for container "donkey", config section "lxc"

Yes, that's a long output. However the focus should obviously go to all the ERROR lines. The first error indicates that the cgroup limits could not be created for this container. Just line above this error shows up a WARN event, which actually shows the reason the cgroup limits could not be created:

Failed to set "cpuset.cpus" to "23-24"

This container was configured to use CPU ids 23 and 24 and therefore 2 vcpus. By looking at htop on that physical machine, one would assume this is correct as 24 CPUs/cores are showing up:

Although this is the way a human counts, it's not how the machine is counting CPUs. cgroups are looking up the number of available cpus under the cgroupfs and under the cpuset.effective_cpus:

root@host ~ # cat /sys/fs/cgroup/cpuset/cpuset.effective_cpus
0-23

The output clearly shows that the count starts at zero (CPU #0) and goes on up to CPU #23. lscpu shows the same way of counting cpus/cores:

root@host ~ # lscpu | grep list
On-line CPU(s) list: 0-23

So the error was caused by the human sitting in front of the keyboard. Now we just need to update the LXC container's config again and set a valid cpu range:

root@host ~ # grep cpu /var/lib/lxc/donkey/config
# cpu and memory limits
lxc.cgroup.cpuset.cpus = 9-10
lxc.cgroup.cpu.shares = 1024

And the container successfully starts:

root@irczsrvp05 ~ # lxc-start -n donkey
root@irczsrvp05 ~ # lxc-ls -f|grep donkey
donkey  RUNNING 1         -      192.168.100.26                  -    false 

Add a comment

Show form to leave a comment

Comments (newest first)

No comments yet.