Source IP incorrect on multi homed systems

[Janne Johansson]

rewriting for the lists, managed to bold some pasted text and hence
get blocked due to html-mails not allowed on list.

Den sön 19 feb. 2023 kl 21:17 skrev Nico Schottelius
<nico.schottelius at ungleich.ch>:
> Janne Johansson <icepic.dz at gmail.com> writes:
> > *) https://en.wiktionary.org/wiki/Chesterton%27s_fence
>
> I am happy to have learned a new principle today, thanks for that.
>
> And to be sure that everyone is on the same page:
>
>     Wireguard should reply by default with the source address that
>     used to be the destination address, but at the moment wireguard is not
>     doing that at the moment.
>
> If anyone disagrees with above statement, please let me know.

I disagree, but perhaps only because that statement is slightly too short.

Let's assume I have two ISPs and hence a multihomed wg peer, with ip
A.x.x.x from isp A, and ip B.x.x.x from isp B. For some reason, this
box has a routing table that says "prefer link A to reach the
internet", but I set up client C to set up wireguard to B.1.2.3 and
client C sends it udp packet with src ip C and dest IP B.x.x.x. Since
UDP is stateless, the "response" from the multihomed server is created
"out of thin air" as a random UDP packet destined for C. We don't feel
it is unrelated to the previous received packet, but from the tcp
stack perspective it is.

The routing table now decides that interface A will be the awesomest
for sending UDP to C, and therefore creates a packet with source ip
A.x.x.x and dest ip C.x.x.x and sends it off. This surprise seems to
be the main issue in this thread. Perhaps we see this multihomed box
as slightly misconfigured as far as wireguard goes, perhaps it should
have posted A.x.x.x instead of B.x.x.x as the wg endpoint to the
client or whatever, but the facts remain.

Now, in your above statement you hope to get everyone to agree on,
this would need to also include "sending it back on interface B, to
the gw used by interface B to ISP B if there is one" or else isp A
might drop the packet as being sent from a "forged" address since it
looks like a fake source ip from isp As perspective. The routing
lookups - before any applied tricks - will look at destination IPs
only and make the decision based on that.

I think the proposed solution, while attractive at first glance, may
be trading one kind of "surprise" behaviour to another where the
interface B might be less useful than A which would explain why the
default route is set to use A. If you look at the many posts on the
internet over many years about "why udp source ip got chosen wrong on
multihomed boxes" you see answers like:
"You either bind(2) to each interface address and manage multiple
sockets, or let the kernel do the implicit source IP assignment with
INADDR_ANY. There is no other way."
( https://stackoverflow.com/questions/3062205/setting-the-source-ip-for-a-udp-socket
, not about vpns and lots older than wireguard)

What this means is that if you have a box where links and interfaces
come and go (usb wifi dongles, tethered cell phones..) then wireguard
now has to make a lot of extra work, trying to keep tabs on what
interfaces exist or not, instead of just binding to port 0 and letting
the kernel handle this by itself in the normal but to some, surprising
way for udp packets.

My gut feeling is that if you have a setup like this example
multihomed peer, you get to do some extra steps, which may include the
aforementioned firewall mark "tricks", use VRFs/Namespaced
interfaces/routing domains, add a specific route to client C ip over
link B, bind wg to a loopback interface or source-nat on outgoing wg
traffic or something along those lines in order to have a wg endpoint
on a less-preferred interface and not cause issues with
stateful-nat-gws at client C.

--
May the most significant bit of your life be positive.