Analysis of packet losses of KVM guest Ubuntu 18.04 on Debian 9 host (interface model type)

Written by Claudio Kuenzler - 1 comments

Published on July 5th 2021 - Listed in Linux Network Virtualization KVM

When creating a new KVM virtual machine with virt-install, it is advised to use the --os-variant parameter. This "fine-tunes" the virtual machine settings for the selected guest OS.

However on a Debian 9 (Stretch) host, the ubuntu18.04 variant could not be used because it does not show up in the list of operating systems (which is used by virt-install in the background). To manually query the list of available os variants, install the libosinfo-bin package and then run osinfo-query os:

The latest Ubuntu entry is Ubuntu 17.10, 18.04 is clearly missing.

As a workaround, the KVM was created without any --os-variant parameter on this Debian 9 host.

Two additional VMs were created on different hosts, running Debian 10 (Buster) with --os-variant ubuntu18.04.

Trouble: Intermittent packet loss

Although the setup of the guest was working without any problems, monitoring of the guest OS in the next hours revealed packet losses. And not just one:

There was even a phase of around 10 minutes with a very high percentage of packet loss.

Looking at the same period of the other two KVM guests showed no packet loss at all, not even a short spike or anything. So what would cause this?

Thanks to the use of monitoring plugin check_netio, the interface statistics are added to the monitoring's performance data. By default this monitoring plugin also includes the interface's DROP and ERR statistics. And interestingly, the graph indeed showed DROPs on the RX (Receiving) statistics, matching the packet loss times:

Inside the affected KVM guest, a manual verification in /proc/net/dev confirmed that there is indeed something wrong and the drop count increased steadily:

root@guest:~# cat /proc/net/dev
Inter-|   Receive                                                | Transmit
face |bytes    packets errs drop fifo frame compressed multicast|bytes    packets errs drop fifo colls carrier compressed
ens3: 220939864 997209    0 60465    0     0          0         0 182069093 1005755    0    0    0 4931718       0          0
    lo: 155226267 589407    0    0    0     0          0         0 155226267 589407    0    0    0     0       0          0

The KVM interface model type

With the knowledge that the KVM guests on the Buster hosts suffered no such problem, the QEMU config of the guests can be checked and compared. And indeed, the difference was quickly spotted in the <interface ...> section!

root@stretch ~ # grep "interface type" -A 5 /etc/libvirt/qemu/guest1.xml
    <interface type='bridge'>
      <mac address='52:54:00:33:81:94'/>
      <source bridge='virbr1'/>
<model type='rtl8139'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </interface>

root@buster ~ # grep "interface type" -A 5 /etc/libvirt/qemu/guest2.xml
    <interface type='bridge'>
      <mac address='52:54:00:ac:92:8e'/>
      <source bridge='virbr1'/>
<model type='virtio'/>
      <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
    </interface>

The KVM guest with the problem had a model type of rtl8139 configured, while the guest created on Debian 10 (Buster) showed a virtio model type.

By manually editing the config (using virsh edit) the model type was changed to virtio and the guest was rebooted.

After this no further packet losses or any other kind of network issue was experienced on that guest anymore.

Add a comment

Show form to leave a comment

Comments (newest first)

Miguel Angel from Murcia (Spain) wrote on Sep 16th, 2022:

I have two Ubuntu VMs for RADIUS and LDAP services, which were showing extremely high network usage statistics with "ip" and "ifconfig": the usage was in the order of Petabytes in a couple of weeks.
However, utilities such as "iptraf" or "tcpdump" showed almost no traffic at all. Another great utility, "nmon" displayed thousands of transmission errors on the virtual interfaces, and it turns out they were configured as "e1000e".

After reading your article, changed them to "virtio", and everything is working like a charm! Thank you, you made my day!

Blog Tags:

AWS Android Ansible Apache Apple Atlassian BSD Backup Bash Bluecoat CMS Chef Cloud Coding Consul Containers CouchDB DB DNS Database Databases Docker ELK Elasticsearch Filebeat FreeBSD Galera Git GlusterFS Grafana Graphics HAProxy HTML Hacks Hardware Icinga Influx Internet Java KVM Kibana Kodi Kubernetes LVM LXC Linux Logstash Mac Macintosh Mail MariaDB Minio MongoDB Monitoring Multimedia MySQL NFS Nagios Network Nginx OSSEC OTRS Office OpenSearch PGSQL PHP Perl Personal PostgreSQL Postgres PowerDNS Proxmox Proxy Python Rancher Rant Redis Roundcube SSL Samba Seafile Security Shell SmartOS Solaris Surveillance Systemd TLS Tomcat Ubuntu Unix VMWare VMware Varnish Virtualization Windows Wireless Wordpress Wyse ZFS Zoneminder