WireGuard IRQ distribution

[Rumen Telbizov]

Hi Roman,

I ran UDP-based iperf and it made no difference.

Thanks for your input

On Mon, Jun 26, 2023 at 7:53 AM Roman Mamedov <rm at romanrm.net> wrote:
>
> On Tue, 9 May 2023 15:17:00 -0700
> Rumen Telbizov <rumen.telbizov at menlosecurity.com> wrote:
>
> > Baseline iperf3 performance over plain VLAN:
> > * Stable 24Gbit/s and 2Mpps
> >
> > bmon:
> >   Gb                      (RX Bits/second)
> > 24.54 .........|.||..|.||.||.||||||..||.||.......................
> > 20.45 .........|||||||||||||||||||||||||||||.....................
> > 16.36 ........||||||||||||||||||||||||||||||.....................
> > 12.27 ........||||||||||||||||||||||||||||||.....................
> > 8.18 ........|||||||||||||||||||||||||||||||.....................
> > 4.09 ::::::::|||||||||||||||||||||||||||||||:::::::::::::::::::::
> >      1   5   10   15   20   25   30   35   40   45   50   55   60
> >    M                     (RX Packets/second)
> > 2.03 .........|.||..|.||.||.||||||..||.||........................
> > 1.69 .........|||||||||||||||||||||||||||||......................
> > 1.35 ........||||||||||||||||||||||||||||||......................
> > 1.01 ........||||||||||||||||||||||||||||||......................
> > 0.68 ........|||||||||||||||||||||||||||||||.....................
> > 0.34 ::::::::|||||||||||||||||||||||||||||||:::::::::::::::::::::
> >      1   5   10   15   20   25   30   35   40   45   50   55   60
> >
> > top:
> > %Cpu0  :  0.0 us,  0.0 sy,  0.0 ni,100.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
> > %Cpu1  :  0.0 us,  0.0 sy,  0.0 ni,100.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
> > %Cpu2  :  1.0 us,  1.0 sy,  0.0 ni, 98.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
> > %Cpu3  :  0.0 us,  0.0 sy,  0.0 ni,100.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
> > %Cpu4  :  0.0 us,  0.0 sy,  0.0 ni,100.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
> > %Cpu5  :  0.0 us,  0.0 sy,  0.0 ni,100.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
> > %Cpu6  :  1.0 us,  0.0 sy,  0.0 ni, 99.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
> > %Cpu7  :  0.0 us,  0.0 sy,  0.0 ni,100.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
> > %Cpu8  :  0.0 us,  0.0 sy,  0.0 ni,100.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
> > %Cpu9  :  1.0 us,  1.0 sy,  0.0 ni, 98.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
> > %Cpu10 :  0.0 us,  0.0 sy,  0.0 ni,100.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
> > %Cpu11 :  0.0 us,  0.9 sy,  0.0 ni, 16.8 id,  0.0 wa,  0.0 hi, 82.2 si,  0.0 st
> > %Cpu12 :  0.0 us, 32.3 sy,  0.0 ni, 65.6 id,  0.0 wa,  0.0 hi,  2.1 si,  0.0 st
> > %Cpu13 :  1.0 us, 36.3 sy,  0.0 ni, 59.8 id,  0.0 wa,  0.0 hi,  2.9 si,  0.0 st
> > %Cpu14 :  0.0 us,  0.0 sy,  0.0 ni,100.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
> > %Cpu15 :  0.0 us,  1.0 sy,  0.0 ni, 99.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
> >
> > The IRQs do pile up behind CPU 11 because iperf3 is single-threaded.
>
> I'm not sure if they pile up because of that, or because of the same reason
> you point in WG's case, the 5-tuple being the same for the single TCP
> connection of iperf3.
>
> Out of interest, maybe you could try iperf3's UDP mode, and apply the same
> port randomization trick as you used for WG, and see if it also makes it see
> the better IRQ distribution, and the speed drop?
>
> --
> With respect,
> Roman