How to use check_esxi_hardware / CIM access without root user on ESXi

Written by Claudio Kuenzler - 5 comments

Published on February 6th 2023 - last updated on November 10th 2023 - Listed in VMware Virtualization Linux

---

The monitoring plugin check_esxi_hardware is a widely used script to monitor the health of physical hardware of a VMware ESXi server. There is one major (security) problem though: The script needs to use the root user to access the CIM server in the background. But there's a technical workaround how to avoid using the root user to access the information from the ESXi CIM server. By using the esxcli command, we can create an (internal) user on the ESXi server with the necessary permissions.

This was successfully tested on ESXi 8 (build 20513097). The same workaround will likely work on other ESXi versions, too.

Enable SSH on ESXi server

Unfortunately the additional user cannot be created (as needed) in the ESXi user interface (as far as I know). Therefore we must be able to access the ESXi server via SSH. If you're not sure how, see here how to enable SSH on ESXi server.

Create additional local user on ESXi server

Once logged into the ESXi server(s), you can create a new local/system user directly on the server. Run the following command, to create an additional local user:

[root@localhost:~] /usr/lib/vmware/busybox/bin/busybox adduser -s /bin/false -G root -h / monitoring
Changing password for monitoring

You can now choose the new password.

A valid password should be a mix of upper and lower case letters,
digits, and other characters.  You can use a 7 character long
password with characters from at least 3 of these 4 classes.
An upper case letter that begins the password and a digit that
ends it do not count towards the number of character classes used.

Alternatively, if no one else can see your terminal now, you can
pick this as your password: "Loosen6Neon=Trust".

Enter new password: **********
Re-type new password: **********
passwd: password updated successfully

In the example above we've created the user "monitoring", added it to the group "root" (necessary) and disabled shell access (/bin/false).

We can now get a list of local (system) accounts on this ESXi server:

[root@localhost:~] esxcli system account list
User ID     Description                                  Shell access
----------  -------------------------------------------  ------------
root        Administrator                                        true
dcui        DCUI User                                            true
vpxuser     VMware VirtualCenter administration account          true
monitoring  Linux User,,,                                       false

The new monitoring user is now listed in the output and we can confirm that shell access is disabled.

Using check_esxi_hardware with non root user

Right after this, the new monitoring user can be used to query the CIM server via the check_esxi_hardware monitoring plugin:

 $ ./check_esxi_hardware.py -H IP.of.esxi.server -U monitoring -P "very.secret"
OK - Server: QEMU Standard PC (Q35 + ICH9, 2009) s/n: None System BIOS: 1.12.0-1 2014-04-01

Note that I have used a virtualized ESXi server running as a KVM virtual machine in this example.

But isn't using root group another security risk?

As mentioned we have ditched the root user for CIM access and created a new local user "monitoring" for this purpose. This new user is in the "root" group however. Without being a member of the "root" group, the access to the CIM server is denied and you would get an authentication error:

$ ./check_esxi_hardware.py -H IP.of.esxi.server -U monitoring -P "very.secret"
UNKNOWN: Authentication Error

But as the new monitoring user was created without shell access (set to /bin/false), the risks are minimized. SSH logins (if SSH is enabled) is not possible for this monitoring user and also access to the ESXi user interface is prevented. An error is shown when trying to log in:

Permission to perform this operation was denied.

Even if someone gets hands on the monitoring credentials, not much can be done to hurt the ESXi server (not via SSH nor by using the vSphere UI anyway). The security risk is therefore greatly minimized, compared to using the root user for monitoring.

How to make this (re-) boot resistant

Thanks to Leon Zhou for noticing the problem, that after a reboot the access for the non-root user doesn't work anymore. See comments.

The steps above work fine as long as the ESXi server is not rebooted. As soon as the ESXi machine gets rebooted, the non-root user doesn't work anymore. Accessing the CIM server results in an Authentication Error once more.

The reason for this is that /etc/security/access.conf is reset after every reboot and if the newly created non-root user is not part of this allow-list, the user won't be able to authenticate.

[root@localhost:~] cat /etc/security/access.conf
# This file is autogenerated and must not be edited.
+:dcui:ALL
+:root:ALL
+:vpxuser:ALL
-:ALL:ALL

Manually adding our non-root user into this file will make the access temporarily work again:

[root@localhost:~] cat /etc/security/access.conf
# This file is autogenerated and must not be edited.
+:dcui:ALL
+:root:ALL
+:vpxuser:ALL
+:monitoring:ALL
-:ALL:ALL

But after another ESXi reboot, the monitoring user is gone from that file. 

To make this (re-) boot resistant, we need to make sure the non-root user is automatically added into /etc/security/access.conf at boot time. To do this, we can use the rc.local scripts to execute commands during system boot.

Use this one-liner to add a new command into /etc/rc.local.d/local.sh, just before the ending "exit 0" line:

[root@localhost:~] sed -i '/^exit/ i\sed -i "/^#/ a\\+:monitoring:ALL" /etc/security/access.conf' /etc/rc.local.d/local.sh

You can now verify /etc/rc.local.d/local.sh, as it should contain a sed command just before the exit line:

[root@localhost:~] cat /etc/rc.local.d/local.sh
#!/bin/sh ++group=host/vim/vmvisor/boot

# local configuration options

# Note: modify at your own risk!  If you do/use anything in this
# script that is not part of a stable API (relying on files to be in
# specific places, specific tools, specific output, etc) there is a
# possibility you will end up with a broken system after patching or
# upgrading.  Changes are not supported unless under direction of
# VMware support.

# Note: This script will not be run when UEFI secure boot is enabled.

sed -i "/^#/ a\+:monitoring:ALL" /etc/security/access.conf
exit 0

This sed command automatically adds our non-root user (monitoring) back into /etc/security/access.conf.

After a ESXi reboot, the monitoring user still has access to the CIM server:

$ ./check_esxi_hardware.py -H IP.of.esxi.server -U monitoring -P "very.secret"
OK - Server: QEMU Standard PC (Q35 + ICH9, 2009) s/n: None System BIOS: 1.12.0-1 2014-04-01

Add a comment

Show form to leave a comment

Comments (newest first)

kiribool from wrote on Nov 30th, 2023:

After adding user by busybox adduser command newusername becomes the owner of / (top level folder) and ssh stops working. Workaround - chown root:root /
Tested on 6.7.0 U3 and 7.

---

ck from Switzerland wrote on Nov 10th, 2023:

Thanks Leon, this works, but only temporarily. /etc/security/access.conf is reset after another system reboot. I have added a workaround into the article which uses the rc.local scripts to execute commands during system boot. This makes the non-root user access survive a reboot.

---

Leon Zhou from wrote on Nov 10th, 2023:

add +:monitor:ALL in /etc/security/access.conf is working, not sure if it is security.

---

ck from Switzerland wrote on Nov 10th, 2023:

Leon, I just started my ESXi 8 test machine again, where I added this user, and I can indeed confirm that I now get an Authentication Error when using the non-root user!

$ ./check_esxi_hardware.py -H myesxi8 -U monitoring -P secret -v
20231110 09:34:49 LCD Status: True
20231110 09:34:49 Chassis Intrusion Status: True
20231110 09:34:49 Connection to https://myesxi8
20231110 09:34:49 Found pywbem version 1.5.0
20231110 09:34:49 Check classe OMC_SMASHFirmwareIdentity
20231110 09:34:49 Global exit set to UNKNOWN
UNKNOWN: Authentication Error

It still works using the root user. That's a real bummer. Thank you for sharing this information!

---

Leon Zhou from wrote on Nov 10th, 2023:

ESXi 8 (build 22380479), only working after add user, not working after reboot.

---