Malformed UTF-8 characters error in Opera DNS UI logs after PowerDNS upgrade

Written by Claudio Kuenzler - 0 comments

Published on March 8th 2022 - Listed in PowerDNS DNS

---

After a PowerDNS primary server was upgraded from 4.2 to 4.5, the Opera DNS UI (an excellent web user interface for PowerDNS) stopped working.

Trying to view the zones (/zones) or a specific user (/users/anyuser) resulted in the following error:

Oops! Something went wrong!

Sorry, but it looks like something needs fixing on the system. The problem has been automatically reported to the administrators, but if you wish, you can also provide additional information about what you were doing that may have triggered the error.

Digging deeper

The Apache error log revealed the following error, whenever /zones or a user detail was requested:

[Tue Mar 08 09:41:56.542504 2022] [php7:notice] [pid 996] [client 10.120.210.40:65267] 1646728916: Pest_Json_Decode: Decoding error: Malformed UTF-8 characters, possibly incorrectly encoded in /var/dns-ui/PestJSON.php:114, referer: http://dnsui.example.com/users
[Tue Mar 08 09:41:56.542537 2022] [php7:notice] [pid 996] [client 10.120.210.40:65267] 1646728916: Stack trace:, referer: http://dnsui.example.com/users
[Tue Mar 08 09:41:56.542544 2022] [php7:notice] [pid 996] [client 10.120.210.40:65267] 1646728916: #0 /var/dns-ui/PestJSON.php(203): PestJSON->jsonDecode('[{"account": "N...'), referer: http://dnsui.example.com/users
[Tue Mar 08 09:41:56.542549 2022] [php7:notice] [pid 996] [client 10.120.210.40:65267] 1646728916: #1 /var/dns-ui/Pest.php(155): PestJSON->processBody('[{"account": "N...'), referer: http://dnsui.example.com/users
[Tue Mar 08 09:41:56.542571 2022] [php7:notice] [pid 996] [client 10.120.210.40:65267] 1646728916: #2 /var/dns-ui/powerdns.php(30): Pest->get('zones', Array, Array), referer: http://dnsui.example.com/users
[Tue Mar 08 09:41:56.542575 2022] [php7:notice] [pid 996] [client 10.120.210.40:65267] 1646728916: #3 /var/dns-ui/model/zonedirectory.php(146): PowerDNS->get('zones'), referer: http://dnsui.example.com/users
[Tue Mar 08 09:41:56.542579 2022] [php7:notice] [pid 996] [client 10.120.210.40:65267] 1646728916: #4 /var/dns-ui/model/user.php(221): ZoneDirectory->list_zones(Array), referer: http://dnsui.example.com/users
[Tue Mar 08 09:41:56.542583 2022] [php7:notice] [pid 996] [client 10.120.210.40:65267] 1646728916: #5 /var/dns-ui/views/zones.php(18): User->list_accessible_zones(Array), referer: http://dnsui.example.com/users
[Tue Mar 08 09:41:56.542587 2022] [php7:notice] [pid 996] [client 10.120.210.40:65267] 1646728916: #6 /var/dns-ui/requesthandler.php(62): require('/var/dns-ui/vie...'), referer: http://dnsui.example.com/users
[Tue Mar 08 09:41:56.542590 2022] [php7:notice] [pid 996] [client 10.120.210.40:65267] 1646728916: #7 /var/dns-ui/public_html/init.php(18): require('/var/dns-ui/req...'), referer: http://dnsui.example.com/users
[Tue Mar 08 09:41:56.542594 2022] [php7:notice] [pid 996] [client 10.120.210.40:65267] 1646728916: #8 {main}, referer: http://dnsui.example.com/users

Looking at the source code of PestJSON.php shows that (received) data is handled by a jsonDecode function (using json_decode). If json_decode doesn't work, the error from the function should be returned. In this case this is the error which can be seen in the log: Malformed UTF-8 characters, possibly incorrectly encoded.

But the question is: What data is causing json_decode to fail? Is it data from the DNS UI internal (PostgreSQL) database? Or data from the PowerDNS API?

Workaround: Downgrade to PowerDNS 4.3

As the DNS UI was still correctly working in PowerDNS 4.2, we downgraded PowerDNS from 4.5 to 4.4. But this still caused the same problems in DNS UI. 

After downgrade from 4.4 to 4.3 the DNS UI worked fine again, no errors showed up neither in the UI nor in the logs.

But as PowerDNS 4.3 is EOL, this is no solution.

Luckily the error could be reproduced in a TEST environment with PowerDNS 4.5, the latest DNS UI (master branch) and after the powerdns MySQL database was imported from the PROD environment.

Using tcpdump to understand the API request

With a TEST environment at hand where the errors can be reproduced, there is enough time to do a couple of try and errors, trying to find the problem. What eventually lead to the right direction was a tcpdump running on the lo (local) interface and capturing the API request while accessing /zones in DNS UI:

 09:42:55.661879 IP 127.0.0.1.52376 > 127.0.0.1.8081: Flags [P.], seq 1:162, ack 1, win 342, options [nop,nop,TS val 1217468079 ecr 1217468078], length 161
E....1@.@...............].jr..Y....V.......
H...H...GET /api/v1/servers/localhost/zones HTTP/1.1
Host: localhost:8081
X-API-Key: secret
Accept: application/json
Content-Type: application/json

So we can see that the PowerDNS API was contacted. This can be reproduced with curl:

root@pdnstest:~# curl -H "X-API-Key: secret" -H "Accept: application/json" -H "Content-Type: application/json" localhost:8081/api/v1/servers/localhost/zones

The result (body) shows a JSON block of all the zones (domains). But already with the first entry some malformed characters could be spotted:

Instead of showing up Zürich with a umlaut, some garbled text was showing up.

To make the JSON response more readable, it can be run through a JSON parser, such as jq:

root@pdnstest:~# curl -s -H "X-API-Key: secret" -H "Accept: application/json" -H "Content-Type: application/json" localhost:8081/api/v1/servers/localhost/zones | jq
[
  {
    "account": "Z.rich Redirect",
    "dnssec": false,
    "edited_serial": 2022030301,
    "id": "example.com.",
    "kind": "Master",
    "last_check": 0,
    "masters": [],
    "name": "example.com.",
    "notified_serial": 2019032104,
    "serial": 2019032104,
    "url": "/api/v1/servers/localhost/zones/example.com."
  },
  {
[...]

Note: The malformed character was replaced with a dot above.

The error message from json_decode now makes sense. This also rules out a problem of DNS UI's own database but it can be pointed to the data coming from PowerDNS (with MySQL backend).

Convert powerdns database from latin1 to utf8

Looking closer at the "powerdns" database in MySQL, shows that all the tables are using the latin1 character encoding:

mysql> SELECT DISTINCT table_name,table_collation FROM information_schema.tables WHERE table_schema = "powerdns" LIMIT 0,15;
+----------------+-------------------+
| table_name     | table_collation   |
+----------------+-------------------+
| comments       | latin1_swedish_ci |
| cryptokeys     | latin1_swedish_ci |
| domainmetadata | latin1_swedish_ci |
| domains        | latin1_swedish_ci |
| records        | latin1_swedish_ci |
| supermasters   | latin1_swedish_ci |
| tsigkeys       | latin1_swedish_ci |
+----------------+-------------------+
7 rows in set (0.00 sec)

The PowerDNS upgrading documentation mentions a change in MySQL character set detection when upgrading to 4.4:

Before 4.4.0, the gmysql backend told the MySQL (or MariaDB) client libraries to automatically detect the client character set and collation, based on the environment locale.
In 4.4.0, the autodetection has been removed. The MySQL/MariaDB client lib will now use its default settings, unless overridden in my.cnf

But could this really be the reason? Was PowerDNS < 4.4 auto-detecting latin1 and the newer versions (>= 4.4) now expect UTF8?

To verify this, the whole database can be dumped, converted from latin1 to utf8 and then imported again:

root@pdnstest:~# mysqldump --add-drop-table powerdns | replace CHARSET=latin1 CHARSET=utf8 | iconv -f latin1 -t utf8 | mysql powerdns

Let's check the response from PowerDNS API with the utf8 data in the database:

root@pdnstest:~# curl -H "X-API-Key: secret" -H "Accept: application/json" -H "Content-Type: application/json" localhost:8081/api/v1/servers/localhost/zones
[{"account": "Zürich Redirect", "dnssec": false, "edited_serial": 2022030301, "id": "example.com.", "kind": "Master", "last_check": 0, "masters": [], "name": "example.com.", "notified_serial": 2019032104, "serial": 2019032104, "url": "/api/v1/servers/localhost/zones/example.com."},
[...]

This time "Zürich" is written correctly!

And right from this moment on, the DNS UI was working again!

Note: All this was also documented in issue #189 on the GitHub repository of dns-ui.

Add a comment

Show form to leave a comment

Comments (newest first)

No comments yet.