WireGuard IRQ distribution

[Roman Mamedov]

On Tue, 9 May 2023 15:17:00 -0700
Rumen Telbizov <rumen.telbizov at menlosecurity.com> wrote:

> Baseline iperf3 performance over plain VLAN:
> * Stable 24Gbit/s and 2Mpps
> 
> bmon:
>   Gb                      (RX Bits/second)
> 24.54 .........|.||..|.||.||.||||||..||.||.......................
> 20.45 .........|||||||||||||||||||||||||||||.....................
> 16.36 ........||||||||||||||||||||||||||||||.....................
> 12.27 ........||||||||||||||||||||||||||||||.....................
> 8.18 ........|||||||||||||||||||||||||||||||.....................
> 4.09 ::::::::|||||||||||||||||||||||||||||||:::::::::::::::::::::
>      1   5   10   15   20   25   30   35   40   45   50   55   60
>    M                     (RX Packets/second)
> 2.03 .........|.||..|.||.||.||||||..||.||........................
> 1.69 .........|||||||||||||||||||||||||||||......................
> 1.35 ........||||||||||||||||||||||||||||||......................
> 1.01 ........||||||||||||||||||||||||||||||......................
> 0.68 ........|||||||||||||||||||||||||||||||.....................
> 0.34 ::::::::|||||||||||||||||||||||||||||||:::::::::::::::::::::
>      1   5   10   15   20   25   30   35   40   45   50   55   60
> 
> top:
> %Cpu0  :  0.0 us,  0.0 sy,  0.0 ni,100.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
> %Cpu1  :  0.0 us,  0.0 sy,  0.0 ni,100.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
> %Cpu2  :  1.0 us,  1.0 sy,  0.0 ni, 98.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
> %Cpu3  :  0.0 us,  0.0 sy,  0.0 ni,100.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
> %Cpu4  :  0.0 us,  0.0 sy,  0.0 ni,100.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
> %Cpu5  :  0.0 us,  0.0 sy,  0.0 ni,100.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
> %Cpu6  :  1.0 us,  0.0 sy,  0.0 ni, 99.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
> %Cpu7  :  0.0 us,  0.0 sy,  0.0 ni,100.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
> %Cpu8  :  0.0 us,  0.0 sy,  0.0 ni,100.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
> %Cpu9  :  1.0 us,  1.0 sy,  0.0 ni, 98.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
> %Cpu10 :  0.0 us,  0.0 sy,  0.0 ni,100.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
> %Cpu11 :  0.0 us,  0.9 sy,  0.0 ni, 16.8 id,  0.0 wa,  0.0 hi, 82.2 si,  0.0 st
> %Cpu12 :  0.0 us, 32.3 sy,  0.0 ni, 65.6 id,  0.0 wa,  0.0 hi,  2.1 si,  0.0 st
> %Cpu13 :  1.0 us, 36.3 sy,  0.0 ni, 59.8 id,  0.0 wa,  0.0 hi,  2.9 si,  0.0 st
> %Cpu14 :  0.0 us,  0.0 sy,  0.0 ni,100.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
> %Cpu15 :  0.0 us,  1.0 sy,  0.0 ni, 99.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
> 
> The IRQs do pile up behind CPU 11 because iperf3 is single-threaded.

I'm not sure if they pile up because of that, or because of the same reason
you point in WG's case, the 5-tuple being the same for the single TCP
connection of iperf3.

Out of interest, maybe you could try iperf3's UDP mode, and apply the same
port randomization trick as you used for WG, and see if it also makes it see
the better IRQ distribution, and the speed drop?

-- 
With respect,
Roman