Src addr code review (Was: Source IP incorrect on multi homed systems)

[dxld at darkboxed.org]

Hi Nico,

On Mon, Feb 20, 2023 at 10:47:36AM +0100, Nico Schottelius wrote:
> Daniel Gröber <dxld at darkboxed.org> writes:
> > Let's look at the code (heavily culled):
> >
> > 	struct flowi4 fl = {
> > 		.saddr = endpoint->src4.s_addr,
> > 	};
> > 	if (cache)
> > 		rt = dst_cache_get_ip4(cache, &fl.saddr);
> 
> What I am wondering is, how did it get into the cache in the first place?

Right so, endpoint->src4 is set in wg_socket_set_peer_endpoint, which is
called either trough through wg_socket_endpoint_from_skb in the handshake
receive code or wg_socket_set_peer_endpoint in the data path.

The _from_skb variant also calls wg_socket_endpoint_from_skb. Here we're
remembering the src addr of the (received) packet in addr4 and the dst addr
we're going to use for sending as src4 as you'd expect:

	endpoint->addr4.sin_family = AF_INET;
	endpoint->addr4.sin_port = udp_hdr(skb)->source;
	endpoint->addr4.sin_addr.s_addr = ip_hdr(skb)->saddr;
	endpoint->src4.s_addr = ip_hdr(skb)->daddr;
	endpoint->src_if4 = skb->skb_iif;

The dst_cache is set just after those zero'ing conditionals we were looking
at before. It's cleared whenever the endpoint/port changes or one of those
cases is hit. Note the dst_cache is only used for data packets, so
handshakes would be unaffected if it was the cause of your woes.

> > @Nico could it perhaps simply be that you're hitting one of these zero'ing
> > cases and that's why it's using regular kernel src addr selection instead
> > of the cached endpoint src4 address?
> 
> That could absolutely be the case. What is funky is that I see the
> problem on two very different systems
>
> Both systems exhibit the behaviour, but maybe it's better to focus on
> System A first, as this seems to be more the "upstream" source.

It is weird indeed, but yeah. One thing at a time.

BTW, what kernel version/distro are we dealing with?

--Daniel