Empty CentOS 7 repository mirrors and how they break CI/CD pipelines

Written by Claudio Kuenzler - 0 comments

Published on August 8th 2024 - Listed in Linux Internet

As CentOS 7 is now officially EOL, the official RPM repository has emptied their packages from the CentOS 7 mirror on July 1st 2024.

Instead of listing all the packages and other repository meta data, the list is now completely empty and only contains a readme file.

The readme file only shows the following information:

This directory (and version of CentOS) is deprecated.

Why keep using a CentOS 7 repo?

So when CentOS 7 is EOL and should be (should have been) replaced, why is there still a need to access the CentOS 7 repo in the first place?

Good and fair question!

Although CentOS 7 is officially end of life since June 30th 2024 (and the yum repos were emptied a day later), the commercial variant, RHEL 7, is still supported until June 30th 2026 with an extended support contract. And if you need to build packages for RHEL 7, one very common way in the past was to use a CentOS machine since both CentOS 7 and RHEL 7 were binary compatible.

CI/CD pipeline broken because of missing packages (404 error)

Now every CI/CD pipeline, which requires to install CentOS 7 packages, runs into a YUM error; some or all packages can't be found and the repository returns a 404 error, even if you use a local mirror:

CI/CD pipeline job error because of missing packages in CentOS 7 repository

The error message clearly shows that the requested file/package was not found on the local repository:

[Errno 14] HTTPS Error 404 - Not Found

In this particular situation, only some packages showed a 404 error. As it turns out this is a local mirror running in Apache JFrog, which, as it's currently defined, has a local cache for remote repositories active. As soon as this cache runs out, all the packages will be gone - and all the packages will throw a 404 on the job output.

What now?

My preferred solution would have been to simply use Rocky Linux and their repositories as an alternative. Unfortunately Rocky Linux was only created later (after CentOS announced their end) and only offer EL8 and EL9 compatible packages/repositories.

A possible workaround could be to use the CentOS Vault, which still offers the CentOS 7 packages - however not in the same repository structure as before.

And then I stumbled across something unexpected: The Linux repositories @ CERN! It turns out their "mirror" is still in a good shape; meaning the original CentOS 7 repository structure and packages are still available!