Fixing broken graphite-web after Linux distribution upgrade

Written by Claudio Kuenzler - 0 comments

Published on May 27th 2024 - Listed in Monitoring Icinga

---

After a Linux distribution upgrade (RHEL 7 to RHEL 8), the Graphite Web user interface stopped working and showed an internal server error instead of the Graphite frontend application:

Note: I, personally, would never go and use Graphite - but on this legacy system, Graphite was (back in the days) the chosen time series database. 

The Graphite installation on that particular host was done using Python pip (not from RPM packages). I was hoping that due to Graphite's installation through pip this would continue to work after the upgrade - but here we are. Let's find out the cause for the error.

Error loading MySQLdb module

As graphite-web is a Python application launched through Apache/HTTPD and the UWSGI module, the application errors can be found in the Apache logs. After a httpd restart, the following errors showed up:

[wsgi:error] [pid 3335:tid 140108262066496] mod_wsgi (pid=3335): Failed to exec Python script file '/opt/graphite/conf/graphite.wsgi'.
 [wsgi:error] [pid 3335:tid 140108262066496] mod_wsgi (pid=3335): Exception occurred processing WSGI script '/opt/graphite/conf/graphite.wsgi'.
 [wsgi:error] [pid 3335:tid 140108262066496] Traceback (most recent call last):
 [wsgi:error] [pid 3335:tid 140108262066496]   File "/opt/graphite/lib/python3.6/site-packages/django/db/backends/mysql/base.py", line 15, in <module>
 [wsgi:error] [pid 3335:tid 140108262066496]     import MySQLdb as Database
 [wsgi:error] [pid 3335:tid 140108262066496]   File "/opt/graphite/lib/python3.6/site-packages/MySQLdb/__init__.py", line 18, in <module>
 [wsgi:error] [pid 3335:tid 140108262066496]     from . import _mysql
 [wsgi:error] [pid 3335:tid 140108262066496] ImportError: libmysqlclient.so.18: cannot open shared object file: No such file or directory
 [wsgi:error] [pid 3335:tid 140108262066496]
 [wsgi:error] [pid 3335:tid 140108262066496] The above exception was the direct cause of the following exception:
 [wsgi:error] [pid 3335:tid 140108262066496]
 [wsgi:error] [pid 3335:tid 140108262066496] Traceback (most recent call last):
 [wsgi:error] [pid 3335:tid 140108262066496]   File "/opt/graphite/conf/graphite.wsgi", line 3, in <module>
 [wsgi:error] [pid 3335:tid 140108262066496]     from graphite.wsgi import application
 [wsgi:error] [pid 3335:tid 140108262066496]   File "/opt/graphite/webapp/graphite/wsgi.py", line 15, in <module>
 [wsgi:error] [pid 3335:tid 140108262066496]     application = get_wsgi_application()
 [wsgi:error] [pid 3335:tid 140108262066496]   File "/opt/graphite/lib/python3.6/site-packages/django/core/wsgi.py", line 12, in get_wsgi_application
 [wsgi:error] [pid 3335:tid 140108262066496]     django.setup(set_prefix=False)
 [wsgi:error] [pid 3335:tid 140108262066496]   File "/opt/graphite/lib/python3.6/site-packages/django/__init__.py", line 24, in setup
 [wsgi:error] [pid 3335:tid 140108262066496]     apps.populate(settings.INSTALLED_APPS)
 [wsgi:error] [pid 3335:tid 140108262066496]   File "/opt/graphite/lib/python3.6/site-packages/django/apps/registry.py", line 112, in populate
 [wsgi:error] [pid 3335:tid 140108262066496]     app_config.import_models()
 [wsgi:error] [pid 3335:tid 140108262066496]   File "/opt/graphite/lib/python3.6/site-packages/django/apps/config.py", line 198, in import_models
 [wsgi:error] [pid 3335:tid 140108262066496]     self.models_module = import_module(models_module_name)
 [wsgi:error] [pid 3335:tid 140108262066496]   File "/opt/graphite/lib64/python3.6/importlib/__init__.py", line 126, in import_module
 [wsgi:error] [pid 3335:tid 140108262066496]     return _bootstrap._gcd_import(name[level:], package, level)
 [wsgi:error] [pid 3335:tid 140108262066496]   File "/opt/graphite/webapp/graphite/account/models.py", line 16, in <module>
 [wsgi:error] [pid 3335:tid 140108262066496]     from django.contrib.auth import models as auth_models
 [wsgi:error] [pid 3335:tid 140108262066496]   File "/opt/graphite/lib/python3.6/site-packages/django/contrib/auth/models.py", line 2, in <module>
 [wsgi:error] [pid 3335:tid 140108262066496]     from django.contrib.auth.base_user import AbstractBaseUser, BaseUserManager
 [wsgi:error] [pid 3335:tid 140108262066496]   File "/opt/graphite/lib/python3.6/site-packages/django/contrib/auth/base_user.py", line 47, in <module>
 [wsgi:error] [pid 3335:tid 140108262066496]     class AbstractBaseUser(models.Model):
 [wsgi:error] [pid 3335:tid 140108262066496]   File "/opt/graphite/lib/python3.6/site-packages/django/db/models/base.py", line 101, in __new__
 [wsgi:error] [pid 3335:tid 140108262066496]     new_class.add_to_class('_meta', Options(meta, app_label))
 [wsgi:error] [pid 3335:tid 140108262066496]   File "/opt/graphite/lib/python3.6/site-packages/django/db/models/base.py", line 305, in add_to_class
 [wsgi:error] [pid 3335:tid 140108262066496]     value.contribute_to_class(cls, name)
 [wsgi:error] [pid 3335:tid 140108262066496]   File "/opt/graphite/lib/python3.6/site-packages/django/db/models/options.py", line 203, in contribute_to_class
 [wsgi:error] [pid 3335:tid 140108262066496]     self.db_table = truncate_name(self.db_table, connection.ops.max_name_length())
 [wsgi:error] [pid 3335:tid 140108262066496]   File "/opt/graphite/lib/python3.6/site-packages/django/db/__init__.py", line 33, in __getattr__
 [wsgi:error] [pid 3335:tid 140108262066496]     return getattr(connections[DEFAULT_DB_ALIAS], item)
 [wsgi:error] [pid 3335:tid 140108262066496]   File "/opt/graphite/lib/python3.6/site-packages/django/db/utils.py", line 202, in __getitem__
 [wsgi:error] [pid 3335:tid 140108262066496]     backend = load_backend(db['ENGINE'])
 [wsgi:error] [pid 3335:tid 140108262066496]   File "/opt/graphite/lib/python3.6/site-packages/django/db/utils.py", line 110, in load_backend
 [wsgi:error] [pid 3335:tid 140108262066496]     return import_module('%s.base' % backend_name)
 [wsgi:error] [pid 3335:tid 140108262066496]   File "/opt/graphite/lib64/python3.6/importlib/__init__.py", line 126, in import_module
 [wsgi:error] [pid 3335:tid 140108262066496]     return _bootstrap._gcd_import(name[level:], package, level)
 [wsgi:error] [pid 3335:tid 140108262066496]   File "/opt/graphite/lib/python3.6/site-packages/django/db/backends/mysql/base.py", line 20, in <module>
 [wsgi:error] [pid 3335:tid 140108262066496]     ) from err
 [wsgi:error] [pid 3335:tid 140108262066496] django.core.exceptions.ImproperlyConfigured: Error loading MySQLdb module.
 [wsgi:error] [pid 3335:tid 140108262066496] Did you install mysqlclient?

There are a lot of lines, and finding the relevant information is not that easy. But here is the most important hint:

• ImportError: libmysqlclient.so.18: cannot open shared object file: No such file or directory

When the application is starting, Django and all it's modules are being loaded. One of these modules is the MySQLdb module (mentioned at the end of the error). According to the log, this module expects a MySQL client library called libmysqlclient.so.18.

However since this Linux was upgraded, the MySQL client libraries have been upgraded, too, and are now available in a higher version:

[root@rhel8 ~]# find / -name libmysqlclient*
/usr/lib64/mysql/libmysqlclient.so.21
/usr/lib64/mysql/libmysqlclient.so.21.2.36
/usr/lib64/libmysqlclient.so
/usr/lib64/libmysqlclient_r.so

Because Graphite was installed in the virtual Python environment before the OS Upgrade, the older (and now missing) MySQL library was used to install and compile the Python pip modules. This means we need to re-install them.

Re-install pip modules in Graphite virtual environment

The Graphite documentation explains hows to install Graphite using pip inside a virtual python environment. The same basic information can be used for an upgrade.

Let's first install the necessary system packages using dnf:

[root@rhel8 ~]# dnf install python36 python36-devel gcc gcc-c++ make libffi-devel

Then first switch into the virtual environment (venv):

[root@rhel8 ~]# su - carbon
[carbon@rhel8 ~]$ cd /opt/graphite
[carbon@rhel8 ~]$ source bin/activate

Now inside this venv, let's check which pip modules are installed (= were installed under RHEL 7):

(graphite) [carbon@rhel8 graphite]$ pip list
Package        Version
-------------- -------
attrs          19.3.0
Automat        0.8.0
cachetools     4.0.0
cairocffi      0.9.0
carbonate      1.1.6
cffi           1.13.2
constantly     15.1.0
Django         2.1.14
django-tagging 0.4.6
hyperlink      19.0.0
idna           2.8
incremental    17.5.0
mod-wsgi       4.6.8
mysqlclient    1.4.5
pip            19.3.1
pycparser      2.19
PyHamcrest     1.9.0
pyparsing      2.4.5
pytz           2019.3
scandir        1.10.0
setuptools     44.0.0
six            1.13.0
Twisted        19.7.0
txAMQP         0.8.2
urllib3        1.25.7
wheel          0.33.6
whisper        1.1.6
whitenoise     4.1.4
zope.interface 4.7.1
WARNING: You are using pip version 19.3.1; however, version 21.3.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.

The output at the end already mentions, that the currently installed modules are outdated.

Let's re-install and upgrade all the Python modules in this venv - except Django (as this would cause a compatibility issue with the old Graphite code):

(graphite) [carbon@rhel8 graphite]$ pip install --upgrade attrs  Automat cachetools cairocffi carbonate cffi constantly django-tagging hyperlink idna incremental mod-wsgi mysqlclient pip pycparser PyHamcrest pyparsing pytz scandir setuptools six Twisted txAMQP urllib3 wheel whisper whitenoise zope.interface

Once this finishes without an error, it's time to test again.

Graphite-Web UI working again

After another httpd restart, the uwsgi errors are gone from the logs:

[root@rhel8 ~]# systemctl restart httpd && tail -f /var/log/httpd/error_log
[mpm_event:notice] [pid 1441:tid 139995743054144] AH00492: caught SIGWINCH, shutting down gracefully
[suexec:notice] [pid 28951:tid 139670281238848] AH01232: suEXEC mechanism enabled (wrapper: /usr/sbin/suexec)
[so:warn] [pid 28951:tid 139670281238848] AH01574: module wsgi_module is already loaded, skipping
[lbmethod_heartbeat:notice] [pid 28951:tid 139670281238848] AH02282: No slotmem from mod_heartmonitor
[mpm_event:notice] [pid 28951:tid 139670281238848] AH00489: Apache/2.4.37 (Red Hat Enterprise Linux) mod_wsgi/4.9.4 Python/3.6 configured -- resuming normal operations
[core:notice] [pid 28951:tid 139670281238848] AH00094: Command line: '/usr/sbin/httpd -D FOREGROUND'

Quick verification in the browser and graphite-web is working again and correctly presenting graphs for recent data (retrieved by go-carbon):

Add a comment

Show form to leave a comment

Comments (newest first)

No comments yet.