Unexpected APT error when trying to enable Ubuntu Pro (ESM) service

Written by Claudio Kuenzler - 1 comments

Published on May 11th 2023 - Listed in Linux Ubuntu Security SSL TLS

---

Ubuntu 18.04 (with code name Bionic) is end of life (EOL) since end of April 2023 and does not receive package or security updates anymore. To fix this, the distribution should be upgraded to a newer Ubuntu LTS version; right now these are Ubuntu 20.04 (Focal) and 22.04 (Jammy).

What if a LTS upgrade is not supported?

But sometimes a distribution upgrade is not possible or is not supported by either hardware or software vendor. For this purpose Canonical, the company behind Ubuntu, offers the "Ubuntu Pro" subscription. This subscription enables the Expanded Security Maintenance (ESM) on your Ubuntu machine. With ESM enabled, you can get an additional 5 years of security updates, without having to upgrade the distribution version.

Unexpected APT error instead of enabled ESM service

After the Ubuntu Pro subscription was enabled (using pro attach <License>) on a Ubuntu 18.04 machine, the status of the different (available) services can be checked:

root@bionic:~# pro status
SERVICE          ENTITLED  STATUS    DESCRIPTION
cc-eal           yes       disabled  Common Criteria EAL2 Provisioning Packages
cis              yes       disabled  Security compliance and audit tools
esm-apps         yes       disabled  Expanded Security Maintenance for Applications
esm-infra        yes       disabled  Expanded Security Maintenance for Infrastructure
fips             yes       disabled  NIST-certified core packages
fips-updates     yes       disabled  NIST-certified core packages with priority security updates
livepatch        yes       disabled  Canonical Livepatch service
ros              yes       disabled  Security Updates for the Robot Operating System
ros-updates      yes       disabled  All Updates for the Robot Operating System

Enable services with: pro enable <service>

                Account: Company
           Subscription: Ubuntu Pro
            Valid until: Mon Apr  1 01:59:59 2024 CEST
Technical support level: essential

But when I tried to enable the ESM services "esm-apps" and "esm-infra", I ran into the following error:

root@bionic:~# pro enable esm-apps
One moment, checking your subscription first
Unexpected APT error. See /var/log/ubuntu-advantage.log

A closer look into the mentioned log file shows problems verifying the server certificate on esm.ubuntu.com:

root@bionic:~# pro enable esm-apps
[...]
2023-05-11 10:37:33,190 - system.py:(389) [DEBUG]: Writing file /var/lib/ubuntu-advantage/private/machine-token.json atomically via tempfile /var/lib/ubuntu-advantage/private/tmppm3c3pxj
2023-05-11 10:37:33,192 - system.py:(389) [DEBUG]: Writing file /var/lib/ubuntu-advantage/machine-token.json atomically via tempfile /var/lib/ubuntu-advantage/tmp_9m6v8q_
2023-05-11 10:37:33,193 - system.py:(362) [DEBUG]: Reading file: /var/lib/ubuntu-advantage/private/machine-token.json
2023-05-11 10:37:33,196 - system.py:(389) [DEBUG]: Writing file /var/lib/ubuntu-advantage/private/machine-id atomically via tempfile /var/lib/ubuntu-advantage/private/tmp_nrhcpuf
2023-05-11 10:37:33,393 - system.py:(512) [DEBUG]: Failed running command '/usr/lib/apt/apt-helper download-file https://bearer:<REDACTED>@esm.ubuntu.com/apps/ubuntu/pool/ /tmp/tmp_i_0spzy/apt-helper-output' [exit(100)]. Message: W: Download is performed unsandboxed as root as file '/tmp/tmp_i_0spzy/apt-helper-output' couldn't be accessed by user '_apt'. - pkgAcquire::Run (13: Permission denied)
E: Failed to fetch https://esm.ubuntu.com/apps/ubuntu/pool/  Certificate verification failed: The certificate is NOT trusted. The certificate issuer is unknown.  Could not handshake: Error in the certificate verification. [IP: 165.225.202.14 9480]
E: Download Failed
 Retrying 3 more times.
[...]

SSL interception on HTTP proxy

All the Ubuntu machines in that particular infrastructure are using a proxy for outgoing HTTP connections. APT is configured to use this proxy. In most cases this does not cause problems, because the Ubuntu base repositories are configured using HTTP (not HTTPS). But the added ESM repositories are using HTTPS. If the proxy uses SSL interception, this replaces the original certificate with a MITM certificate from the proxy. And (rightfully so), APT is complaining about the unknown certificate issuer.

There are two ways to solve this:

1) Install the certificate chain from the MITM/Proxy certificate on the Ubuntu machine

2) Disable SSL interception for the domain (esm.ubuntu.com) on the proxy

We've decided for option 2 in this case as we trust the Ubuntu servers. After this config change was done, the ESM services could be enabled:

root@bionic:~# pro enable esm-apps
One moment, checking your subscription first
Updating package lists
Ubuntu Pro: ESM Apps enabled

root@bionic:~# pro enable esm-infra
One moment, checking your subscription first
Updating package lists

Seems that it worked this time, yay!

Using the pro status command again to verify the service status:

root@bionic:~# pro status
SERVICE          ENTITLED  STATUS    DESCRIPTION
cc-eal           yes       disabled  Common Criteria EAL2 Provisioning Packages
cis              yes       disabled  Security compliance and audit tools
esm-apps         yes       enabled   Expanded Security Maintenance for Applications
esm-infra        yes       enabled   Expanded Security Maintenance for Infrastructure
fips             yes       disabled  NIST-certified core packages
fips-updates     yes       disabled  NIST-certified core packages with priority security updates
livepatch        yes       disabled  Canonical Livepatch service
ros              yes       disabled  Security Updates for the Robot Operating System
ros-updates      yes       disabled  All Updates for the Robot Operating System

Enable services with: pro enable <service>

                Account: Company
           Subscription: Ubuntu Pro
            Valid until: Mon Apr  1 01:59:59 2024 CEST
Technical support level: essential

The wanted services are now enabled and package updates can be installed from the enabled ESM repositories.

Add a comment

Show form to leave a comment

Comments (newest first)

Marty Hillman from Minnesota wrote on Oct 17th, 2024:

We have the exact same error occurring on new server builds, but are not using an outbound proxy. Before the "certificate is NOT trusted" message, we see the following.

Failed running command '/usr/lib/apt/apt-helper download-file https://bearer:@esm.ubuntu.com/apps/ubuntu/pool/ /tmp/tmp4hyl1tgg/apt-helper-output' [exit(100)]. Message: W: Download is performed unsandboxed as root as file '/tmp/tmp4hyl1tgg/apt-helper-output' couldn't be accessed by user '_apt'. - pkgAcquire::Run (13: Permission denied)

---