[RFC] Replace WireGuard AllowedIPs with IP route attribute

[Kyle Rose]

Daniel,

I attempted several times to send the following message to the mailing
list nearly two months ago, but they all landed in moderation for some
reason and were silently dropped after a few days. You are not the
only one who wants this functionality for mesh VPNs. I note the email
you subsequently replied to (by Bernd) also didn't show up on the
list, suggesting I am not the only one whose messages are being
inexplicably jailed.

Kyle

====
I really like the straightforward configurability of Wireguard
out-of-the-box. It was astonishingly easy to configure a mesh to
replace my previous hub-and-spoke OpenVPN setup. Thank you for making
this easy.

That said, I'd like the ability to use Linux's policy routing engine
to allow for more complex packet flows across the VPN that are
currently incompatible with Wireguard's internal packet handling. For
example, let's say I have 4 nodes and want to be able to use each of
the nodes as the default gateway for different types of flows.
Modifying the sender side to respect a route's gateway is
straightforward:

--- a/drivers/net/wireguard/allowe
dips.c
+++ b/drivers/net/wireguard/allowe
dips.c
@@ -6,6 +6,8 @@
 #include "allowedips.h"
 #include "peer.h"

+#include <net/route.h>
+
 enum { MAX_ALLOWEDIPS_BITS = 128 };

 static struct kmem_cache *node_cache;
@@ -356,10 +358,18 @@ int wg_allowedips_read_node(struct
allowedips_node *node, u8 ip[16], u8 *cidr)
 struct wg_peer *wg_allowedips_lookup_dst(stru
ct allowedips *table,
                                         struct sk_buff *skb)
 {
-       if (skb->protocol == htons(ETH_P_IP))
-               return lookup(table->root4, 32, &ip_hdr(skb)->daddr);
-       else if (skb->protocol == htons(ETH_P_IPV6))
-               return lookup(table->root6, 128, &ipv6_hdr(skb)->daddr);
+       struct rtable *rt = skb_rtable(skb);
+       if (rt->rt_uses_gateway) {
+               if (rt->rt_gw_family == AF_INET)
+                       return lookup(table->root4, 32, &rt->rt_gw4);
+               else if (rt->rt_gw_family == AF_INET6)
+                       return lookup(table->root6, 128, &rt->rt_gw6);
+       } else {
+               if (skb->protocol == htons(ETH_P_IP))
+                       return lookup(table->root4, 32, &ip_hdr(skb)->daddr);
+               else if (skb->protocol == htons(ETH_P_IPV6))
+                       return lookup(table->root6, 128, &ipv6_hdr(skb)->daddr);
+       }
        return NULL;
 }

The problem is that reply packets will be dropped via the source
address check for all but the peer with the default route listed in
AllowedIPs. The way the trie code works means a highly-invasive change
would be needed to allow for multiple peers to be associated with a
given prefix: I suspect any further complication (not to mention
possible additional data structure bloat) is undesirable, and anyway I
am looking to bypass most of the complexity created by Wireguard's
parallel packet routing infrastructure and instead leverage the far
more flexible Linux policy routing engine.

At this point, what I'd like to do is be able to skip the source
address check by configuration, instead relying on rp_filter and
firewall rules to reject bogus or unwanted packets. With such a config
knob, AllowedIPs would be used only for selecting the destination peer
based on the packet daddr or the route's gateway. For a mesh like I
described above, I would configure only a gateway IP for each peer in
AllowedIPs and use the policy routing engine for all other packet
routing behavior.

I appreciate that Wireguard works the way it does most likely because
routing is configured differently across the wide variety of devices
it supports (and in some cases may be unavailable to users), so I
don't think the AllowedIPs source address check should be removed by
default; but it would be nice if I could turn it off and rely on other
mechanisms in the kernel that would allow for more flexibility.


On Sat, Aug 19, 2023 at 10:05 AM Daniel Gröber <dxld at darkboxed.org> wrote:
>
> Hi wireguard, birds, and babelers,
>
> tl;dr I want to add a new Linux route attribute (think "via $wgpeer") to
> supplement wireguard's internal AllowedIPs logic for both routing and
> source address filtering.
>
> I've been pondering how to better integrate wireguard into dynamic routing
> daemons, particularly BIRD and babeld. Essentially we want to be able to
> dynamically add/remove AllowedIPs depending on current reachability and/or
> link quality stats.
>
> Looking at the wg netlink API I see two major efficiency/scalability
> problems: 1) there is no way to be notified of changes in AllowedIPs made
> by other processes meaning we have to do periodic scans and 2) a peer's
> AllowedIPs set can only be replaced wholesale, not modified
> incrementally. This is problematic as "someone" might, in the worst case,
> want to install an entire internet routing table's worth of AllowedIPs and
> the set will likely change frequently. FYI: The IPv4 table has ~1M entries
> at present, yikes.
>
> Assuming external AllowedIPs changes are infrequent occationally dumping
> them all to keep a consistent view of the state shouldn't be too much of an
> issue as long as the netlink interface is performant enoug, so I'm going to
> concentrate on the add/remove API for now.
>
> Instead of doing the obvious thing and adding a more efficient incremental
> AllowedIPs netlink interface I figure why not just add a route attribute to
> select a target wg peer on a device. That way we could not only save memory
> (no separate AllowedIPs trie) but also simplify routing daemon
> implementation considerably.
>
> This would mirror how on ethernet you can have `dev eth0 via $router_ip`.
> I'm still reviewing the net/ code to find the best way to do this, but I'm
> thinking either a new RTA_WGPEER, like: `default dev wg0 via-wgpeer
> $peer_pubkey` or perhaps re-using RTA_VIA and keying off a statically
> configured AllowedIP addresses.
>
> To start I'd make this an opt-in replacement for our usual AllowedIPs
> logic, making sure to only activate it if any via* RTAs are active on a
> particular device, but if it proves to work well I don't see why we
> couldn't adapt the netlink code to maintain AllowedIPs using this RTA (but
> invisible to userspace) to re-use the same code and get rid of allowedips.c
> altogether. That's assuming this ends up being less code overall or perhaps
> more performant.
>
> Happy to hear your thoughts,
> --Daniel