Terrible libvirt Network Performance

I have recently started using Linux for virtualization on my desktop, and encountered terrible network performance between the host and (FreeBSD) guest.

Here are some representative numbers, obtained using iperf:

# Host IP: 192.168.3.254
# Guest IP: 192.168.3.1

# Host -> guest
# $ guest ~> iperf -s
# $ host  ~> iperf -c 192.168.3.1
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-11.7 sec  3.00 MBytes  2.14 Mbits/sec

# Guest to host
# $ host  ~> iperf -s
# $ guest ~> iperf -c 192.168.3.254
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.0 sec  1.63 GBytes  1.14 Gbits/sec

This is a truly abysmal result. Remember, these numbers are for transfer rates between a host and a virtualized guest -- we are not even hitting my network.

What could the problem be?

    $ virsh domiflist net.anserinae.installation03
    Interface  Type       Source     Model       MAC
    -------------------------------------------------------
    vnet0      network    default    rtl8139     52:54:00:cd:58:29

Wait what? An rtl8139? From if_rl.c on FreeBSD:

 * The RealTek 8139 PCI NIC redefines the meaning of 'low end.' This is
 * probably the worst PCI ethernet controller ever made, with the possible
 * exception of the FEAST chip made by SMC. The 8139 supports bus-master
 * DMA, but it has a terrible interface that nullifies any performance
 * gains that bus-master DMA usually offers.

 [snip]

 * It's impossible given this rotten design to really achieve decent
 * performance at 100Mbps, unless you happen to have a 400Mhz PII or
 * some equally overmuscled CPU to drive it.

Let's use something slightly less terrible for our guest's NIC:

$ virsh edit net.anserinae.installation03

Change this...

<interface type='bridge'>
  [random output]
  <model type='rtl8139'/>
</inteface>

...to:

<interface type='bridge'>
  [random output]
  <model type='virtio'/>
</inteface>

Restart the virtual machine:

$ virsh reset net.anserinae.installation03

...and retry the performance tests again:

# Host -> guest
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-11.7 sec  52.9 MBytes  45.4 Gbits/sec

# Guest -> host
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.0 sec  59.7 GBytes  51.3 Gbits/sec

Performance is, conservatively speaking, improved by three orders of magnitude.

I haven't tested rtl8139 performance on any other guest operating systems, but my gut feeling here is to say that the libvirt maintainers could have chosen a slightly less garbage NIC as the default option for newly created virtual machines.