CPU round-robin and isolated cores

[Charles-François Natali]

Hi!

We've run into an issue where wireguard doesn't play nice with
isolated cores (`isolcpus` kernel parameter).

Basically we use `isolcpus` to isolate cores and explicitly bind our
low-latency processes to those cores, in order to minimize latency due
to the kernel and userspace.

It worked great until we started using wireguard, in particular it
seems to be due to the way work is allocated to the workqueues created here:
https://github.com/torvalds/linux/blob/ae085d7f9365de7da27ab5c0d16b12d51ea7fca9/drivers/net/wireguard/device.c#L335

I'm not familiar with the wireguard code at all so might be missing
something, but looking at e.g.
https://github.com/torvalds/linux/blob/ae085d7f9365de7da27ab5c0d16b12d51ea7fca9/drivers/net/wireguard/receive.c#L575
and https://github.com/torvalds/linux/blob/ae085d7f9365de7da27ab5c0d16b12d51ea7fca9/drivers/net/wireguard/queueing.h#L176
it seems that the RX path uses round-robin to dispatch the
packets to all online CPUs, including isolated ones:

```
void wg_packet_receive(struct wg_device *wg, struct sk_buff *skb)
{
[...]
    /* Then we queue it up in the device queue, which consumes the
    * packet as soon as it can.
    */
    cpu = wg_cpumask_next_online(next_cpu);
    if (unlikely(ptr_ring_produce_bh(&device_queue->ring, skb)))
        return -EPIPE;
    queue_work_on(cpu, wq, &per_cpu_ptr(device_queue->worker, cpu)->work);
    return 0;
}
```

Where `wg_cpumask_next_online` is defined like this:
```
static inline int wg_cpumask_next_online(int *next)
{
    int cpu = *next;

    while (unlikely(!cpumask_test_cpu(cpu, cpu_online_mask)))
        cpu = cpumask_next(cpu, cpu_online_mask) % nr_cpumask_bits;
    *next = cpumask_next(cpu, cpu_online_mask) % nr_cpumask_bits;
    return cpu;
}
```

It's a problem for us because it causes significant latency, see e.g.
this ftrace output showing a kworker - bound to an isolated core - spend over
240usec inside wg_packet_decrypt_worker - we've seen much higher, up to
500usec or even more:

```
kworker/47:1-2373323 [047] 243644.756405: funcgraph_entry: |
process_one_work() {
    kworker/47:1-2373323 [047] 243644.756406: funcgraph_entry: |
wg_packet_decrypt_worker() {
[...]
    kworker/47:1-2373323 [047] 243644.756647: funcgraph_exit: 0.591 us | }
    kworker/47:1-2373323 [047] 243644.756647: funcgraph_exit: ! 242.655 us | }
```

If it was for example a physical NIC, typically what we'd do would
be to set IRQ affinity to avoid those isolated cores, which would also
avoid running the corresponding softirqs on those cores, avoiding such
latency.

However it seems that there's currently no way to tell wireguard to
avoid those cores.

I was wondering if it would make sense for wireguard to ignore
isolated cores to avoid this kind of issue. As far as I can tell it should
be a matter of replacing usages of `cpu_online_mask` by
`housekeeping_cpumask(HK_TYPE_DOMAIN)` or even
`housekeeping_cpumask(HK_TYPE_DOMAIN | HK_TYPE_WQ)`.

We could potentially run with a patched kernel but would very much
prefer using an upstream fix if that's acceptable.

Thanks in advance!

Charles