Source IP incorrect on multi homed systems

Sun Feb 19 22:28:48 UTC 2023

Actually in my case (I'm not the originator of this thread), I don't run 
BGP.  But I do have both site-site and mobile-site clients.  Much 
simpler environment, but same issue.

I do understand UDP.

As I've noted, DNS UDP has the same issue, and an RFC was issued to 
clarify that responses MUST come from the address on which a query is 
received.

WG isn't quite the same, as it isn't a request/response protocol.  But 
it is a flow between two endpoints, and NAT/firewalls will open a 
pinhole for incoming packets when they see an outbound packet.

One of the nice things about WG is that except for this issue, it has no 
dependencies on custom routing (or anything but UDP) and "just works".  
It should "just work" on multihomed hosts, without handstands, BGP 
routing, different ports, and the like.  It also needs to work where 
it's not feasible to layer on work-arounds, such as VPSs where you don't 
get to pick your kernel...or your firewall.

Picking stable endpoint addresses would make the traffic look like the 
kind of flow that these middleboxes recognize, and things would "just 
work".

On 19-Feb-23 13:25, John Lauro wrote:
> I think the ip route with src would work, but only as a short lived 
> work around.  The problem with that is if dealing with dynamic routes 
> is it could go a way when a link is down and then come back and the 
> src setting would be lost.  You would need the bgp software to add the 
> src.
>
> UDP is connectionless.  Sending back out the same as it's coming in 
> isn't strictly the same.  The streams are not attached the same as 
> they would be with TCP on nginx or a reply with icmp. You should be 
> able to whitelist the udp port on the NAT devices, as it shouldn't use 
> state info.
>
> I am not sure if you are attempting to do site to site or client to 
> server/site and which end has the NAT (or both). What I do for site to 
> site is use a different port for each connection and have a separate 
> BGP connection for each possible connection (ie: different one for 
> different network providers).  Have a full mesh with 8 sites and upto 
> 3 providers per site.
>
> That said, you probably have floating IPs on the client side, and 
> don't want to lock in a single IP on the multi-homed server side?  You 
> could nat the incoming IPs on the border from an internal IP and then 
> then lock to a single private IP on the wireguard server for in/out 
> and that border nat would force the reply back to the same gateway it 
> came in from.
>
> I know, you don't want work arounds, just want to mention it's not the 
> same as comparing a single stream to something that handles routing 
> though it.  As you are doing bgp and redundant routes I assume you 
> also reset rp_filter on all nat/wireguard/routers so the routers will 
> allow packets to come from different sources.
>
> On Sun, Feb 19, 2023 at 12:07 PM tlhackque <tlhackque at yahoo.com> wrote:
>
>     FWIW, while clever, I don't think that iptables mark solves all
>     cases.
>     E.g., consider an interface with multiple addresses, where a packet
>     comes in on a secondary address.  The proposed rule would send it out
>     the right interface, but still with the wrong (primary) address
>     picked
>     from the interface...
>
>     With IPv6 it's common to assign an address to a service rather than a
>     host so services can move easily.  So multiple addresses per
>     interface
>     are the rule, not the exception.
>
>     I do the same with IPv4 inside addresses, though these days public
>     IPv4
>     addresses are scarce enough that it's not common for public IPs.  It
>     amounts to the same issue - the NAT tracking is stateful.
>
>     Trying to work around this with routing seems like a maze of twisty
>     passages - so I agree that the right solution is for WG to respond
>     from
>     the address that receives a packet.
>
>     On 19-Feb-23 11:32, David Kerr wrote:
>     > Without getting into the debate of whether wireguard is acting
>     > correctly or not, I think there is a possible workaround.
>     >
>     > 1. In the iptables mangle table PREROUTING, match the incoming
>     > interface and destination address and --set-xmark a firewall MARK
>     > unique to this interface/destination
>     > 2. Create a new ip route table that sets the default route to go out
>     > on the interface with the source address you want (same as
>     destination
>     > address in iptables)
>     > 3. Create a new ip rule that sends all packets with firewall
>     mark set
>     > in iptables to the routing table you just created
>     >
>     > Repeat above for each interface/address you need to mangle, with a
>     > unique firewall mark and routing table for each.
>     >
>     > It may be necessary to use CONNMARK in PREROUTING and OUTPUT to
>     > --restore_mark.  I can't remember if this is needed or not, its
>     been a
>     > while since I configured iptables with this.
>     >
>     > This should ensure that any packet that comes into an
>     > interface/address is replied to from the same interface/address.
>     >
>     > David
>     >
>     >
>     > On Sun, Feb 19, 2023 at 9:44 AM Christoph
>     Loesch<wireguard-mail at chil.at> wrote:
>     >> Hi,
>     >>
>     >> I don't think no one wants to fix it, there are several users
>     having this issue. I rather guess no one could find a suitable
>     solution to fix it.
>     >>
>     >> @Nico: did you try to delete the affected route and add it
>     again with the correct source IP ?
>     >>
>     >> as I mentioned it
>     inhttps://lists.zx2c4.com/pipermail/wireguard/2021-November/007324.html
>     <http://lists.zx2c4.com/pipermail/wireguard/2021-November/007324.html>
>     >>
>     >> ip route del <NET>
>     >> ip route add <NET> dev <ALIAS_DEV> src <SRC_IP>
>     >>
>     >> This way I was able to (at least temporary) fix this issue on
>     multi homed systems.
>     >>
>     >> Kind regards,
>     >> Christoph
>     >>
>     >> Am 19.02.2023 um 13:13 schrieb Nico Schottelius:
>     >>> Hey Sebastian,
>     >>>
>     >>> Sebastian Hyrwall<sh at keff.org>  writes:
>     >>>
>     >>>> It is kinda. It's been mentioned multiple times over the
>     years but no one seems to want to fix it. Atleast you should be
>     able to specify bind/src ip in the
>     >>>> config. I gave up WG because of it. Wasn't accepted by my
>     projects security policy since src ip could not be configured.
>     >>>>
>     >>>> There is an unofficial patch however,
>     >>>>
>     >>>>
>     https://github.com/torvalds/linux/commit/5fa98082093344c86345f9f63305cae9d5f9f281
>     >>> the binding is somewhat related to this issue and I was
>     looking for that
>     >>> feature some time ago, too. While it is correlated and I would
>     really
>     >>> appreciate binding support, I am not sure whether the linked
>     patch does
>     >>> actually fix the problem I am seeing in multi homed devices.
>     >>>
>     >>> As long as wireguard does not reply with the same IP address
>     it was
>     >>> contacted with, packets will get dropped on stateful
>     firewalls, because
>     >>> the returning packet does not match the state session database.
>     >>>
>     >>> Best regards,
>     >>>
>     >>> Nico
>     >>>
>     >>> --
>     >>> Sustainable and modern Infrastructures by ungleich.ch
>     <http://ungleich.ch>
>

-------------- next part --------------
A non-text attachment was scrubbed...
Name: OpenPGP_signature
Type: application/pgp-signature
Size: 840 bytes
Desc: OpenPGP digital signature
URL: <http://lists.zx2c4.com/pipermail/wireguard/attachments/20230219/bfc99915/attachment.sig>