Matomo import_logs.py script running into json.decoder.JSONDecodeError error

Written by Claudio Kuenzler - 0 comments

Published on September 5th 2024 - Listed in Linux Internet

---

The Matomo (previously known as Piwik) log importer ran into the following error while trying to parse yesterday's access log of my site:

ck@matomo:~# python3 /var/www/piwik.example.com/misc/log-analytics/import_logs.py --idsite=1 --url=http://piwik.example.com --token-auth=XXX --enable-http-errors --enable-http-redirects --enable-static --enable-bots --enable-reverse-dns --recorders=4 --log-format-name=ncsa_extended /var/log/apache2/www.claudiokuenzler.com.access.log.1
Traceback (most recent call last):
  File "/var/www/piwik.example.com/misc/log-analytics/import_logs.py", line 1723, in _call_api
    return json.loads(res)
  File "/usr/lib/python3.9/json/__init__.py", line 346, in loads
    return _default_decoder.decode(s)
  File "/usr/lib/python3.9/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/lib/python3.9/json/decoder.py", line 355, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/var/www/piwik.example.com/misc/log-analytics/import_logs.py", line 2823, in <module>
    resolver = config.get_resolver()
  File "/var/www/piwik.example.com/misc/log-analytics/import_logs.py", line 1266, in get_resolver
    return StaticResolver(self.options.site_id)
  File "/var/www/piwik.example.com/misc/log-analytics/import_logs.py", line 1798, in __init__
    site = matomo.call_api(
  File "/var/www/piwik.example.com/misc/log-analytics/import_logs.py", line 1782, in call_api
    return self._call_wrapper(self._call_api, None, None, method, **kwargs)
  File "/var/www/piwik.example.com/misc/log-analytics/import_logs.py", line 1734, in _call_wrapper
    response = func(*args, **kwargs)
  File "/var/www/piwik.example.com/misc/log-analytics/import_logs.py", line 1725, in _call_api
    raise urllib.error.URLError('Matomo returned an invalid response: ' + res.decode("utf-8") )
AttributeError: 'str' object has no attribute 'decode'

http to https redirect

The error message itself is rather cryptic than clear. The only error - kind of - pointing to the actual problem was:

    raise urllib.error.URLError('Matomo returned an invalid response: ' + res.decode("utf-8") )

It would have been helpful that the actual response code would be shown in the error message. As the --url parameter tried to connect to a http URL, a 301 (permanent redirect) response code would have been sent by the web server.

Once I changed the --url parameter to use the https domain, the import worked again:

ck@matomo:~# python3 /var/www/piwik.example.com/misc/log-analytics/import_logs.py --idsite=1 --url=https://piwik.example.com --token-auth=xxx --enable-http-errors --enable-http-redirects --enable-static --enable-bots --enable-reverse-dns --recorders=4 --log-format-name=ncsa_extended /var/log/apache2/www.claudiokuenzler.com.access.log.1
0 lines parsed, 0 lines recorded, 0 records/sec (avg), 0 records/sec (current)
Parsing log /var/log/apache2/www.claudiokuenzler.com.access.log.1...
3200 lines parsed, 0 lines recorded, 0 records/sec (avg), 0 records/sec (current)
3200 lines parsed, 649 lines recorded, 324 records/sec (avg), 649 records/sec (current)
4000 lines parsed, 1876 lines recorded, 624 records/sec (avg), 1227 records/sec (current)
4800 lines parsed, 2447 lines recorded, 611 records/sec (avg), 571 records/sec (current)
[...]

But why did it work before?

This Matomo server was recently migrated and the Matomo installation was upgraded right afterwards. The Matomo system checks revealed it was not able to communicate with itself (https://piwik.examle.com) using curl_exec.

The reason was that I previously used Matomo to communicate with itself via localhost, setting a DNS entry in /etc/hosts pointing to its own local address. However in my setup the TLS certificate is placed on a reverse proxy, not on the Matomo server itself; a local https connection would therefore not work. After I removed the /etc/hosts entry, the Matomo system checks resolved the DNS to itself to use the public IP, went through the reverse proxy and could talk with itself this way.

But then I simply forgot to adjust the shell scripts which run the import_logs.py script on a daily basis - they still contained the http URL assuming the /etc/hosts entry was still there.

Add a comment

Show form to leave a comment

Comments (newest first)

No comments yet.