From adrian.sev at gmail.com Wed Jun 2 06:36:14 2021 From: adrian.sev at gmail.com (Adrian Sevcenco) Date: Wed, 2 Jun 2021 09:36:14 +0300 Subject: tunnel no longer established :: how to debug? Message-ID: Hi! After a change of connection point (the same ISP but different location, but with the same wireless, and end-points) the tunnel no longer is established. Nothing in the wg configuration is changed ... from the "client" machine to the "server" end-point i can do: root at hal: ~ # nc -u -vz sev.HHHHHH 43333 Ncat: Version 7.80 ( https://nmap.org/ncat ) Ncat: Connected to HHH.HHH.HHH.HHH:43333. Ncat: UDP packet sent successfully Ncat: 1 bytes sent, 0 bytes received in 2.01 seconds. How can i debug further to see why the connection is not established? Thanks a lot! Adrian From adrian.sev at gmail.com Wed Jun 2 07:23:12 2021 From: adrian.sev at gmail.com (Adrian Sevcenco) Date: Wed, 2 Jun 2021 10:23:12 +0300 Subject: tunnel no longer established :: how to debug? - solved In-Reply-To: References: Message-ID: On 6/2/21 9:36 AM, Adrian Sevcenco wrote: > Hi! After a change of connection point (the same ISP but different location, but with the same wireless, and end-points) > the tunnel no longer is established. Nothing in the wg configuration is changed ... > from the "client" machine to the "server" end-point i can do: > root at hal: ~ # nc -u -vz sev.HHHHHH 43333 > Ncat: Version 7.80 ( https://nmap.org/ncat ) > Ncat: Connected to HHH.HHH.HHH.HHH:43333. > Ncat: UDP packet sent successfully > Ncat: 1 bytes sent, 0 bytes received in 2.01 seconds. > > How can i debug further to see why the connection is not established? i apologize for the noise! the problem was a typo on the server configuration where a peer was defined also with it's endpoint (dynamic) dns name. this worked fine as long that name was resolving correctly but as soon as this failed the connection could no longer be established. Thanks! Adrian From wireguard at clodo.it Wed Jun 2 12:11:12 2021 From: wireguard at clodo.it (Fabrizio Carimati) Date: Wed, 2 Jun 2021 14:11:12 +0200 Subject: Windows embeddable-dll-service, avoid default routes Message-ID: I need to open a tunnel without route all traffic, only specific multiple subnets. Under Linux or macOS, i use 0.0.0.0/0 as AllowedIPs and after i create my specific routes (i don't use wg-quick). It's correct that routes are outside the scope of WireGuard, imho. But embeddable-dll-service create a 0.0.0.0/0 metric 0 route, how i can avoid it (apart remove it after...)? Thanks From Jason at zx2c4.com Wed Jun 2 14:45:34 2021 From: Jason at zx2c4.com (Jason A. Donenfeld) Date: Wed, 2 Jun 2021 16:45:34 +0200 Subject: Windows embeddable-dll-service, avoid default routes In-Reply-To: References: Message-ID: If you don't want 0.0.0.0/0, then don't specify it in your AllowedIPs. Instead put your more specific routes there. From arinc.unal at gmail.com Thu Jun 3 06:43:04 2021 From: arinc.unal at gmail.com (=?UTF-8?B?QXLEsW7DpyDDnE5BTA==?=) Date: Thu, 3 Jun 2021 09:43:04 +0300 Subject: WireGuard responses a Handshake Initiation packet with another Handshake Initiation instead of Handshake Response Message-ID: I tried wg syncconf wg0 <(wg-quick strip wg0) without success. Only restarting wireguard by systemctl restart wg-quick at wg0 makes it respond with a Handshake Response packet. This happens really rarely and especially on OpenWrt 19.07 as a client. Other clients can handshake properly while the server fails to complete the handshake for the OpenWrt device. Packet Captures on boths sides are in the attachments. Server: 5.9.0-0.bpo.5-amd64 #1 SMP Debian 5.9.15-1~bpo10+1 (2020-12-31) x86_64 GNU/Linux wireguard-tools v1.0.20210223 - https://git.zx2c4.com/wireguard-tools/ wireguard-dkms/buster-backports,now 1.0.20210219-1~bpo10+1 Client: 4.14.221 #0 SMP Mon Feb 15 15:22:37 2021 armv7l GNU/Linux wireguard-tools v1.0.20191226 - https://git.zx2c4.com/wireguard-tools/ kmod-wireguard 4.14.221+1.0.20200611-2 Ar?n? ?. -------------- next part -------------- A non-text attachment was scrubbed... Name: server.pcapng Type: application/octet-stream Size: 9796 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: client.pcapng Type: application/octet-stream Size: 5888 bytes Desc: not available URL: From mail at 53c70r.de Thu Jun 3 19:30:39 2021 From: mail at 53c70r.de (Silvan Nagl) Date: Thu, 3 Jun 2021 21:30:39 +0200 Subject: WireGuard responses a Handshake Initiation packet with another Handshake Initiation instead of Handshake Response In-Reply-To: References: Message-ID: <852812b5-9210-5c70-7cef-d673967d568a@53c70r.de> Reproducible: Client_a <-> Gateway_a <-> Internet <-> Server_b When Gateway_a changes its WAN-IP Client_a loses connection. Capture files show the state after the IP change. On 6/3/21 8:43 AM, Ar?n? ?NAL wrote: > I tried wg syncconf wg0 <(wg-quick strip wg0) without success. > > Only restarting wireguard by systemctl restart wg-quick at wg0 makes it > respond with a Handshake Response packet. > > This happens really rarely and especially on OpenWrt 19.07 as a client. > > Other clients can handshake properly while the server fails to > complete the handshake for the OpenWrt device. > > Packet Captures on boths sides are in the attachments. > > Server: > 5.9.0-0.bpo.5-amd64 #1 SMP Debian 5.9.15-1~bpo10+1 (2020-12-31) x86_64 GNU/Linux > wireguard-tools v1.0.20210223 - https://git.zx2c4.com/wireguard-tools/ > wireguard-dkms/buster-backports,now 1.0.20210219-1~bpo10+1 > > Client: > 4.14.221 #0 SMP Mon Feb 15 15:22:37 2021 armv7l GNU/Linux > wireguard-tools v1.0.20191226 - https://git.zx2c4.com/wireguard-tools/ > kmod-wireguard 4.14.221+1.0.20200611-2 > > Ar?n? ?. -------------- next part -------------- A non-text attachment was scrubbed... Name: server.pcap Type: application/vnd.tcpdump.pcap Size: 1054 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: client.pcap Type: application/vnd.tcpdump.pcap Size: 1260 bytes Desc: not available URL: From mail at 53c70r.de Thu Jun 3 19:40:42 2021 From: mail at 53c70r.de (Silvan Nagl) Date: Thu, 3 Jun 2021 21:40:42 +0200 Subject: WireGuard responses a Handshake Initiation packet with another Handshake Initiation instead of Handshake Response In-Reply-To: <852812b5-9210-5c70-7cef-d673967d568a@53c70r.de> References: <852812b5-9210-5c70-7cef-d673967d568a@53c70r.de> Message-ID: actually it was reproducible on my 4w old setup. after updating both it ain't reproducible anymore. openwrt: Quintus Build at 2021.05.23 / LuCI openwrt-21.02 branch git-21.136.64332-53c572a fedora_server: 5.12.8-300.fc34.x86_64 On 03.06.2021 21:30, Silvan Nagl wrote: > Reproducible: > > Client_a <-> Gateway_a <-> Internet <-> Server_b > > When Gateway_a changes its WAN-IP Client_a loses connection. > Capture files show the state after the IP change. > > On 6/3/21 8:43 AM, Ar?n? ?NAL wrote: >> I tried wg syncconf wg0 <(wg-quick strip wg0) without success. >> >> Only restarting wireguard by systemctl restart wg-quick at wg0 makes it >> respond with a Handshake Response packet. >> >> This happens really rarely and especially on OpenWrt 19.07 as a client. >> >> Other clients can handshake properly while the server fails to >> complete the handshake for the OpenWrt device. >> >> Packet Captures on boths sides are in the attachments. >> >> Server: >> 5.9.0-0.bpo.5-amd64 #1 SMP Debian 5.9.15-1~bpo10+1 (2020-12-31) x86_64 GNU/Linux >> wireguard-tools v1.0.20210223 - https://git.zx2c4.com/wireguard-tools/ >> wireguard-dkms/buster-backports,now 1.0.20210219-1~bpo10+1 >> >> Client: >> 4.14.221 #0 SMP Mon Feb 15 15:22:37 2021 armv7l GNU/Linux >> wireguard-tools v1.0.20191226 - https://git.zx2c4.com/wireguard-tools/ >> kmod-wireguard 4.14.221+1.0.20200611-2 >> >> Ar?n? ?. From Jason at zx2c4.com Sun Jun 6 09:13:36 2021 From: Jason at zx2c4.com (Jason A. Donenfeld) Date: Sun, 6 Jun 2021 11:13:36 +0200 Subject: potentially disallowing IP fragmentation on wg packets, and handling routing loops better Message-ID: Hi, WireGuard is an encrypted point-to-multipoint tunnel, where onion layering of packets via a single interface or multiple is a useful feature. This makes handling routing loops very hard to manage and detect. I'm considering changing and simplifying loop mitigation to a different strategy, but not without some discussion of its implications. Specifically the change would be to not allow IP fragmentation of the encrypted UDP packets. This way, in the case of a loop, eventually the packet size exceeds MTU, and it gets dropped: dumb and effective. Depending on how this discussion goes, a compromise would be to not allow fragmentation, but only for forwarded and kernel-generated packets, not not for locally generated userspace packets. That's more complex and I don't like it as much as just disallowing IP fragmentation all together. Pros: - It solves the routing loop problem very simply. - Usually when people are fragmenting packets like that, things become very, very slow anyway, and it'd be better to just stop working entirely, so that people adjust their MTU. - Is anybody actually relying on this? Cons: - Maybe people are running wireguard-over-gre-over-vxlan-over-l2tp-over-pppoe-over-god-knows-what-else, and this reduces the MTU to below 1280, yet they still want to put IPv6 through wireguard, and are willing to accept the performance implications. - Some people don't know how to fix their MTUs, and breaking rather than just becoming really slow isn't the best outcome there, maybe. - Maybe people are relying on this? Before anybody asks: we're not going to add a knob for this, simply by virtue of this being a decision with pros and cons. Please don't bring that up. I'd be very interested in opinions about this. Are there additional pros and cons? I know the matter has come up a few times on the list, mostly with people _wanting_ fragmentation (I've CCd a few people from those threads - Roman, I expect you to vigorously argue the pro-fragmentation stance ;-). but I'm not convinced the outcome of those threads was correct, other than, "yea, that's easy enough to enable." But on the other hand, maybe the cons are real enough we should rethink this. Please let me know thoughts and ideas. Thanks, Jason From Jason at zx2c4.com Sun Jun 6 09:36:53 2021 From: Jason at zx2c4.com (Jason A. Donenfeld) Date: Sun, 06 Jun 2021 11:36:53 +0200 Subject: [ANNOUNCE] wireguard-linux-compat v1.0.20210606 released Message-ID: -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 Hello, A new version, v1.0.20210606, of the backported WireGuard kernel module for 3.10 <= Linux <= 5.5.y has been tagged in the git repository. == Changes == * netns: make sure rp_filter is disabled on vethc * kbuild: do not use -O3 * global: use synchronize_net rather than synchronize_rcu * peer: allocate in kmem_cache * allowedips: initialize list head in selftest * allowedips: remove nodes in O(1) * allowedips: allocate nodes in kmem_cache * allowedips: free empty intermediate nodes when removing single node * allowedips: add missing __rcu annotation to satisfy sparse * qemu: increase default dmesg log size These are from the 5.13 pull request: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=6fd815bb1ecc5d3cd99a31e0393fba0be517ed04 * qemu: add disgusting hacks for RHEL 8 Red Hat's latest kernel no longer compiles correctly, so we have to add all sorts of hacks to fix it. I sent them some patches 6 months ago but they never picked them up. This release contains commits from: Jason A. Donenfeld. As always, the source is available at https://git.zx2c4.com/wireguard-linux-compat/ and information about the project is available at https://www.wireguard.com/ . This version is available in compressed tarball form here: https://git.zx2c4.com/wireguard-linux-compat/snapshot/wireguard-linux-compat-1.0.20210606.tar.xz SHA2-256: 3f5d990006e6eabfd692d925ec314fff2c5ee7dcdb869a6510d579acfdd84ec0 A PGP signature of that file decompressed is available here: https://git.zx2c4.com/wireguard-linux-compat/snapshot/wireguard-linux-compat-1.0.20210606.tar.asc Signing key: AB9942E6D4A4CFC3412620A749FC7012A5DE03AE Remember to unxz the tarball before verifying the signature. If you're a package maintainer, please bump your package version. If you're a user, the WireGuard team welcomes any and all feedback on this latest version. Finally, WireGuard development thrives on donations. By popular demand, we have a webpage for this: https://www.wireguard.com/donations/ Thank you, Jason Donenfeld -----BEGIN PGP SIGNATURE----- iQJEBAEBCAAuFiEEq5lC5tSkz8NBJiCnSfxwEqXeA64FAmC8lxsQHGphc29uQHp4 MmM0LmNvbQAKCRBJ/HASpd4DrmCUD/oCNMLbpF21NhWXh8gAlfiN2jwoRblm5Ibq mO1VTLlBLf7Cr7PufHJ5sEWfTeqMoLIDx8XWXZdbgUwfMhPDHcNXITH1n7RQmBQt A8b2jWUBBqqZipsEL+Sq/EBpMrVtfwNUM1mcuLsQ1j2ol2Mm6vmn42fTVjz4hvD9 NPWlLzHsnwt6q7t4/HiODWtJATXNMP+CxF7GPF5rSLL+zOAFkFSRoMupO0qTUCIi pCjQoTYD5d5GD/5xVc12uaElYN9AJq7q7J78yAtvZ/BCubHQ80MYwXA5vwQ65sQI dVLZ1vWPU3r4xe2xLewsvvCq318ojO54Zwn9yld0ZDpAJtgcDA0lb13iWjTh8x9o Z7IcqUAT66tpZe1FA1YY00T2+SpqQoCNBhiMpUe8dd7Aqy//1N8y64WdGIPF9SGH +7Eb1X2aaXebacksLbKcfkZbuXwtdExKMpY/ppHceMz3UoacEehf7F1azZmNfo+S l+Fs6v7fduPpSOetz97HoPsgunjpffs8EfV2XuwBe8dqRXlLs97CHCjGR1ct5jua ZOAkOh8UErsSGJc+imEQmRysXsqxIcTACAvCnijZ38d65/Tw0twouSW1EMl8yu8i rG9GVwvwuK+8Aq9me8nXyd3Ik7pUyKtKs2Hs51/ruJQLfgYPt6lqz8k+boEl2rNn F/yp2xPdtQ== =JJZr -----END PGP SIGNATURE----- From diggest at gmail.com Sun Jun 6 10:39:21 2021 From: diggest at gmail.com (Vasili Pupkin) Date: Sun, 6 Jun 2021 13:39:21 +0300 Subject: potentially disallowing IP fragmentation on wg packets, and handling routing loops better In-Reply-To: References: Message-ID: <5b6e244e-017a-1e8a-3778-179fa785f236@gmail.com> Hi, I've dig into the subject two years ago and only vague remember details. As far as I can recall there was a time when WireGuard set DF flag by default and there were two issues: 1) for security reasons WireGuard doesn't issue ICMP fragmentation required response in the unencrypted channel if an encrypted packed didn't fit and was dropped 2) there is no way client can tell the server of MTU limitation it has on its side Combining the two we have a situation in a chained wireguard VPN setup when MTU size is misconfigured on the server and the remote host wouldn't get any icmp to help with its PMTUD algorithm. The client can still set MSS in its TCP connection though. Again sorry if I missed or messed something, it was long ago and I don't remember details. On 06.06.2021 12:13, Jason A. Donenfeld wrote: > Hi, > > WireGuard is an encrypted point-to-multipoint tunnel, where onion > layering of packets via a single interface or multiple is a useful > feature. This makes handling routing loops very hard to manage and > detect. I'm considering changing and simplifying loop mitigation to a > different strategy, but not without some discussion of its > implications. > > Specifically the change would be to not allow IP fragmentation of the > encrypted UDP packets. This way, in the case of a loop, eventually the > packet size exceeds MTU, and it gets dropped: dumb and effective. > Depending on how this discussion goes, a compromise would be to not > allow fragmentation, but only for forwarded and kernel-generated > packets, not not for locally generated userspace packets. That's more > complex and I don't like it as much as just disallowing IP > fragmentation all together. > > Pros: > - It solves the routing loop problem very simply. > - Usually when people are fragmenting packets like that, things become > very, very slow anyway, and it'd be better to just stop working > entirely, so that people adjust their MTU. > - Is anybody actually relying on this? > > Cons: > - Maybe people are running > wireguard-over-gre-over-vxlan-over-l2tp-over-pppoe-over-god-knows-what-else, > and this reduces the MTU to below 1280, yet they still want to put > IPv6 through wireguard, and are willing to accept the performance > implications. > - Some people don't know how to fix their MTUs, and breaking rather > than just becoming really slow isn't the best outcome there, maybe. > - Maybe people are relying on this? > > Before anybody asks: we're not going to add a knob for this, simply by > virtue of this being a decision with pros and cons. Please don't bring > that up. > > I'd be very interested in opinions about this. Are there additional > pros and cons? I know the matter has come up a few times on the list, > mostly with people _wanting_ fragmentation (I've CCd a few people from > those threads - Roman, I expect you to vigorously argue the > pro-fragmentation stance ;-). but I'm not convinced the outcome of > those threads was correct, other than, "yea, that's easy enough to > enable." But on the other hand, maybe the cons are real enough we > should rethink this. > > Please let me know thoughts and ideas. > > Thanks, > Jason > From nico.schottelius at ungleich.ch Sun Jun 6 09:32:44 2021 From: nico.schottelius at ungleich.ch (Nico Schottelius) Date: Sun, 06 Jun 2021 11:32:44 +0200 Subject: potentially disallowing IP fragmentation on wg packets, and handling routing loops better In-Reply-To: References: Message-ID: <87v96ribib.fsf@ungleich.ch> Hello, so given that fragmentation is disallowed the PMTU discovery always needs to work and the wireguard MTU needs to be correctly adjusted. Speaking of a DC situation, I think this might be tricky. Imagine the following situation: - endhost A has an MTU of 9k. PMTU 9k. wg 8920. - the path changes, the PMTU reduces to 1.5k (this is something we see happening from time to time) - How is the wg MTU adjusted in this situation? And to clarify: with disallowing IP frag, you are obviously only referring to the outter transport. Within the tunnels, IPv6 and IPv6 packets can still be fragmented, so application operation is not really affected. Interesting approach, I am not really sure if realisticly feasible, especially when thinking about long range/low bandwidth media where you'd basically say "wg cannot do IPv6 on these mediums". Satelite systems should probably work fine, I am more concerned about mesh networks, in which wg is quite popular already. Cheers, Nico -- Sustainable and modern Infrastructures by ungleich.ch From peter at fiberdirekt.se Sun Jun 6 11:14:16 2021 From: peter at fiberdirekt.se (Peter Linder) Date: Sun, 6 Jun 2021 13:14:16 +0200 Subject: potentially disallowing IP fragmentation on wg packets, and handling routing loops better In-Reply-To: References: Message-ID: This would break things for me. We're doing a lot of L2 over L3 site to site stuff and we are using wireguard as the outer layer. Inner layer is vxlan or l2tpv3. In particular, people connect lots of stuff with no regard for MTU. For some things it's also very hard to change so we just assume people don't. Since the L3 network typically has the same MTU as the inner L2 network, we need fragmentation. There is no practical way to be able to tell hosts on the L2 network about the limited mtu, for all we know they don't even run IP.... It really does work without a hassle, it is not very very slow at all. Performance is down perhaps by a factor of 3 compared to setting a smaller MTU/MSS, but we can still push 350mbit/s with an atom 2ghz cpu, and around 800mbit/s with a xeon cpu, with fragmentation for most packets. This is one case where wireguard really works well! IMHO, having wireguard generating fragmentable packets adds a lot to its usefulness. With that said, it's not the end of the world for me as I can just compile my own but I'd rather not :-) On 2021-06-06 11:13, Jason A. Donenfeld wrote: > Hi, > > WireGuard is an encrypted point-to-multipoint tunnel, where onion > layering of packets via a single interface or multiple is a useful > feature. This makes handling routing loops very hard to manage and > detect. I'm considering changing and simplifying loop mitigation to a > different strategy, but not without some discussion of its > implications. > > Specifically the change would be to not allow IP fragmentation of the > encrypted UDP packets. This way, in the case of a loop, eventually the > packet size exceeds MTU, and it gets dropped: dumb and effective. > Depending on how this discussion goes, a compromise would be to not > allow fragmentation, but only for forwarded and kernel-generated > packets, not not for locally generated userspace packets. That's more > complex and I don't like it as much as just disallowing IP > fragmentation all together. > > Pros: > - It solves the routing loop problem very simply. > - Usually when people are fragmenting packets like that, things become > very, very slow anyway, and it'd be better to just stop working > entirely, so that people adjust their MTU. > - Is anybody actually relying on this? > > Cons: > - Maybe people are running > wireguard-over-gre-over-vxlan-over-l2tp-over-pppoe-over-god-knows-what-else, > and this reduces the MTU to below 1280, yet they still want to put > IPv6 through wireguard, and are willing to accept the performance > implications. > - Some people don't know how to fix their MTUs, and breaking rather > than just becoming really slow isn't the best outcome there, maybe. > - Maybe people are relying on this? > > Before anybody asks: we're not going to add a knob for this, simply by > virtue of this being a decision with pros and cons. Please don't bring > that up. > > I'd be very interested in opinions about this. Are there additional > pros and cons? I know the matter has come up a few times on the list, > mostly with people _wanting_ fragmentation (I've CCd a few people from > those threads - Roman, I expect you to vigorously argue the > pro-fragmentation stance ;-). but I'm not convinced the outcome of > those threads was correct, other than, "yea, that's easy enough to > enable." But on the other hand, maybe the cons are real enough we > should rethink this. > > Please let me know thoughts and ideas. > > Thanks, > Jason From dev at marvingaube.de Sun Jun 6 09:24:20 2021 From: dev at marvingaube.de (Marvin Gaube) Date: Sun, 6 Jun 2021 11:24:20 +0200 Subject: [PATCH wireguard-linux] Basic support for binding the transport socket to a device Message-ID: <20210606092420.85239-1-dev@marvingaube.de> This patch depends on da5095d052860baa7fe2932fb1209628dd3e3813 from udp_tunnel module, and allows to bind the transport socket to a specific interface. With this patch, it is possible to use wireguard with VRFs: The transport uses a separate "WAN" VRF, cleanly isolating Local/VPN and WAN Routing. The userspace API is designed to transmit the device index of the device to listen on. Listening on a device does only work if the socketdev_index is set/changed before the socket is brought up. Signed-off-by: Marvin Gaube --- drivers/net/wireguard/device.h | 1 + drivers/net/wireguard/netlink.c | 8 +++++++- drivers/net/wireguard/socket.c | 10 ++++++++++ include/uapi/linux/wireguard.h | 3 +++ 4 files changed, 21 insertions(+), 1 deletion(-) diff --git a/drivers/net/wireguard/device.h b/drivers/net/wireguard/device.h index 854bc3d97150..a08f0893f687 100644 --- a/drivers/net/wireguard/device.h +++ b/drivers/net/wireguard/device.h @@ -56,6 +56,7 @@ struct wg_device { struct list_head device_list, peer_list; unsigned int num_peers, device_update_gen; u32 fwmark; + u32 socketdev_index; u16 incoming_port; }; diff --git a/drivers/net/wireguard/netlink.c b/drivers/net/wireguard/netlink.c index d0f3b6d7f408..c594a7063038 100644 --- a/drivers/net/wireguard/netlink.c +++ b/drivers/net/wireguard/netlink.c @@ -27,7 +27,8 @@ static const struct nla_policy device_policy[WGDEVICE_A_MAX + 1] = { [WGDEVICE_A_FLAGS] = { .type = NLA_U32 }, [WGDEVICE_A_LISTEN_PORT] = { .type = NLA_U16 }, [WGDEVICE_A_FWMARK] = { .type = NLA_U32 }, - [WGDEVICE_A_PEERS] = { .type = NLA_NESTED } + [WGDEVICE_A_PEERS] = { .type = NLA_NESTED }, + [WGDEVICE_A_SOCKETDEV_INDEX] = { .type = NLA_U32 } }; static const struct nla_policy peer_policy[WGPEER_A_MAX + 1] = { @@ -232,6 +233,7 @@ static int wg_get_device_dump(struct sk_buff *skb, struct netlink_callback *cb) if (nla_put_u16(skb, WGDEVICE_A_LISTEN_PORT, wg->incoming_port) || nla_put_u32(skb, WGDEVICE_A_FWMARK, wg->fwmark) || + nla_put_u32(skb, WGDEVICE_A_SOCKETDEV_INDEX, wg->socketdev_index) || nla_put_u32(skb, WGDEVICE_A_IFINDEX, wg->dev->ifindex) || nla_put_string(skb, WGDEVICE_A_IFNAME, wg->dev->name)) goto out; @@ -538,6 +540,10 @@ static int wg_set_device(struct sk_buff *skb, struct genl_info *info) goto out; } + if (info->attrs[WGDEVICE_A_SOCKETDEV_INDEX]) { + wg->socketdev_index = nla_get_u32(info->attrs[WGDEVICE_A_SOCKETDEV_INDEX]); + } + if (flags & WGDEVICE_F_REPLACE_PEERS) wg_peer_remove_all(wg); diff --git a/drivers/net/wireguard/socket.c b/drivers/net/wireguard/socket.c index d9ad850daa79..885d8a890079 100644 --- a/drivers/net/wireguard/socket.c +++ b/drivers/net/wireguard/socket.c @@ -370,8 +370,18 @@ int wg_socket_init(struct wg_device *wg, u16 port) .use_udp6_rx_checksums = true, .ipv6_v6only = true }; + if (wg->socketdev_index > 0) { + port6.bind_ifindex = wg->socketdev_index; + } else { + port6.bind_ifindex = 0; + } #endif + if (wg->socketdev_index > 0) { + port4.bind_ifindex = wg->socketdev_index; + } else { + port4.bind_ifindex = 0; + } rcu_read_lock(); net = rcu_dereference(wg->creating_net); net = net ? maybe_get_net(net) : NULL; diff --git a/include/uapi/linux/wireguard.h b/include/uapi/linux/wireguard.h index ae88be14c947..52b1bd422f50 100644 --- a/include/uapi/linux/wireguard.h +++ b/include/uapi/linux/wireguard.h @@ -29,6 +29,7 @@ * WGDEVICE_A_PUBLIC_KEY: NLA_EXACT_LEN, len WG_KEY_LEN * WGDEVICE_A_LISTEN_PORT: NLA_U16 * WGDEVICE_A_FWMARK: NLA_U32 + * WGDEVICE_A_SOCKETDEV_INDEX: NLA_U32 * WGDEVICE_A_PEERS: NLA_NESTED * 0: NLA_NESTED * WGPEER_A_PUBLIC_KEY: NLA_EXACT_LEN, len WG_KEY_LEN @@ -83,6 +84,7 @@ * WGDEVICE_A_PRIVATE_KEY: len WG_KEY_LEN, all zeros to remove * WGDEVICE_A_LISTEN_PORT: NLA_U16, 0 to choose randomly * WGDEVICE_A_FWMARK: NLA_U32, 0 to disable + * WGDEVICE_A_SOCKETDEV_INDEX: NLA_U32, 0 to disable * WGDEVICE_A_PEERS: NLA_NESTED * 0: NLA_NESTED * WGPEER_A_PUBLIC_KEY: len WG_KEY_LEN @@ -157,6 +159,7 @@ enum wgdevice_attribute { WGDEVICE_A_LISTEN_PORT, WGDEVICE_A_FWMARK, WGDEVICE_A_PEERS, + WGDEVICE_A_SOCKETDEV_INDEX, __WGDEVICE_A_LAST }; #define WGDEVICE_A_MAX (__WGDEVICE_A_LAST - 1) -- 2.25.1 From Jason at zx2c4.com Sun Jun 6 13:38:15 2021 From: Jason at zx2c4.com (Jason A. Donenfeld) Date: Sun, 06 Jun 2021 15:38:15 +0200 Subject: [ANNOUNCE] wireguard-freebsd snapshot v0.0.20210606 is available Message-ID: -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 Hi, An experimental snapshot, v0.0.20210606, of WireGuard for FreeBSD has been been tagged in the git repository. At this time this code is new, unvetted, possibly buggy, and should be considered "experimental". It might contain security issues. We gladly welcome your testing and bug reports, but do keep in mind that this code is new, so some caution should be exercised at the moment for using it in mission critical environments. == Changes == * netns: use `exit 0` for early exit * ci: add a Cirrus-CI config file to build + smoke test * if_wg: pass back result of selftests and enable in CI * netns: trim test to working parts and rework jail logic * netns: account for FreeBSD 12 quirks * netns: use massive datagrams * if_wg: do not crash if deiniting before vnet is up * ci: test on 12.1 and 12.2 We now have the vnet CI running on every commit on 12.1, 12.2, and 13.0, and so a decent amount of work went into enabling this and tidying up the tests we run. * global: destroy rwlocks and mtxs * global: replace rwlock with mtx if never rlocked This uses some fancy SmPL to simplify our lock types when possible. * compat: account for lack of CSUM_SND_TAG on ?12.2 * if_wg: add braces for 12.1 compiler warning Usual compat work. This snapshot contains commits from: Jason A. Donenfeld and Ed Maste. The source repository is available at the usual location: git clone https://git.zx2c4.com/wireguard-freebsd This snapshot is available in compressed tarball form: https://git.zx2c4.com/wireguard-freebsd/snapshot/wireguard-freebsd-0.0.20210606.tar.xz SHA2-256: d7501666030619e4ee86e68d2b9fd6a718be6541be83bf726414b821c60383d0 Thank you, Jason Donenfeld -----BEGIN PGP SIGNATURE----- iQJEBAEBCAAuFiEEq5lC5tSkz8NBJiCnSfxwEqXeA64FAmC8z7UQHGphc29uQHp4 MmM0LmNvbQAKCRBJ/HASpd4DrpTQD/9VenrT33AbdVf688t11dyKD6S/aZ0XLsF8 Qvjpv8tJuLOOHqSm1d7c6UKdsDrc9okbAvQvHoiX/VhU9CxBKzQoK74iTPiO+bL4 Y7Z+fe+25vzjyzJmW8KhPymgQzfhw2JW7KwjtwXqh4NnZ3zKFoaY/9hWBnZAIF2o pTrjBE6jL5W6smFDyHsa6L0f9hfOJ7PUvNRj9Tym9VQiGxY0Q3z7N845n2gvgwJD J2UKTugz12zdVt/sKSpqLaLrbGaonqY+10SdLVUNO19+6iIx7hRBPw+BKTakUVGl aqji3kM7nsszWIt9yEKzdmG62oIMfZg0524yU3ZW9f38TyN2x/5bNoaRcHxNvcA/ SJ+25mVsSKNiNvVho89e6NxSK9tqywkXCbVd62p6KyOKdJSOvsvuT5LhPzlA6RMz Oli8bpZHQucxNHFzNdnuojo44/G9RlKriCKqMtPY2BH+LJOswzFG071HcWn785hR 96wDkxPvL4QqD/7tHssyPna33Ees/IRSlnYTXaZ5L9wwBTilI0XnKO/X+whpm4jj 21rXfXEXlCbe9tA90H6tQS/sHcn9Ezj2+5MEgCxhZ6yRjeibDZvKR8kCuIcXJxAi /Y1iF5wyVvWSndNKruvUV7t4+w39WzzxaLNokZ3UZYysmLBvwbf+Ddl5yVUxtEkr HmA9Tocw9g== =eYnb -----END PGP SIGNATURE----- From rcmcdonald91 at gmail.com Sun Jun 6 14:27:59 2021 From: rcmcdonald91 at gmail.com (Christian McDonald) Date: Sun, 6 Jun 2021 10:27:59 -0400 Subject: Certain private keys being mangled by wg on FreeBSD Message-ID: Hi Jason, I've got an interesting case here. I've got what appears to be a few private keys provided by Mullvad that are being mangled by wg(8) on FreeBSD12.2 (pfSense 2.5.1+). Private key ```Lm0BA4UyPilHH5qnXCfr6Lw8ynecOPor88tljLy3ALk=``` ... becomes ```KG0BA4UyPilHH5qnXCfr6Lw8ynecOPor88tljLy3AHk=``` on wg showconf another example is ```zQwldrChCYP3P2fZeTtHmj3c/88hxM+TaLIq1mJ28ZY=``` turns into ```yAwldrChCYP3P2fZeTtHmj3c/88hxM+TaLIq1mJ28VY=``` I must have just gotten lucky with the keys Mullvad provided to me, as those worked and continue to work fine for me. ref : https://github.com/theonemcdonald/pfSense-pkg-WireGuard/issues/105 Best, -- R. Christian McDonald E: rcmcdonald91 at gmail.com From Jason at zx2c4.com Sun Jun 6 15:09:01 2021 From: Jason at zx2c4.com (Jason A. Donenfeld) Date: Sun, 6 Jun 2021 17:09:01 +0200 Subject: Certain private keys being mangled by wg on FreeBSD In-Reply-To: References: Message-ID: It looks like whatever is generating those private keys is not clamping them. Specifically, all private keys should undergo this transformation: key[0] &= 248; key[31] = (key[31] & 127) | 64; In your case, your `Lm` prefix (first byte: 0x2c) is being anded with 248, and thus turns into KG (first byte: 0x28). The kernel properly clamps the keys on input, though, in case generators forget to clamp them. So what you're seeing is correct behavior. From rcmcdonald91 at gmail.com Sun Jun 6 15:59:21 2021 From: rcmcdonald91 at gmail.com (Christian McDonald) Date: Sun, 6 Jun 2021 11:59:21 -0400 Subject: Certain private keys being mangled by wg on FreeBSD In-Reply-To: References: Message-ID: Would it not be better for wg to just fail outright instead of transforming a poorly generated key entered by a user, regardless of where the key came from? Especially if that problematic key passes the regex validation that was provided in another thread in this email list? If not, what would be an appropriate solution to catch situations like this and in turn alert users? This seems like it could be a larger discussion on interoperability, especially when dealing with keys that are being generated by VPN providers. Granted, this certainly isn?t my area of expertise. Though, the behavior is just unexpected (and confusing) from an end user perspective. On Sun, Jun 6, 2021 at 11:09 AM Jason A. Donenfeld wrote: > > It looks like whatever is generating those private keys is not > clamping them. Specifically, all private keys should undergo this > transformation: > > key[0] &= 248; > key[31] = (key[31] & 127) | 64; > > In your case, your `Lm` prefix (first byte: 0x2c) is being anded with > 248, and thus turns into KG (first byte: 0x28). > > The kernel properly clamps the keys on input, though, in case > generators forget to clamp them. So what you're seeing is correct > behavior. -- R. Christian McDonald M: (616) 856-9291 E: rcmcdonald91 at gmail.com From Jason at zx2c4.com Sun Jun 6 16:21:34 2021 From: Jason at zx2c4.com (Jason A. Donenfeld) Date: Sun, 6 Jun 2021 18:21:34 +0200 Subject: Certain private keys being mangled by wg on FreeBSD In-Reply-To: References: Message-ID: On 6/6/21, Christian McDonald wrote: > Would it not be better for wg to just fail outright instead of > transforming a poorly generated key entered by a user, regardless of > where the key came from? Especially if that problematic key passes the > regex validation that was provided in another thread in this email > list? No, it would not be better. There is nothing wrong with using those keys. They're not "poorly generated" or "problematic" or dangerous in the least. This is only a concern with your UI. The kernel is doing the correct thing -- clamping keys -- and displaying an unambiguous identifier to the user: the key that it will actually be using. I suspect the best thing to do for your UI would be to hide private (and preshared) keys, and only show public keys, unless explicitly exported into a config file. This not only reduces potential confusion with this issue, but mitigates another potential footgun down the line. It's also what wg(8)'s show command does by default (while showconf will export all). From rm at romanrm.net Sun Jun 6 19:03:18 2021 From: rm at romanrm.net (Roman Mamedov) Date: Mon, 7 Jun 2021 00:03:18 +0500 Subject: potentially disallowing IP fragmentation on wg packets, and handling routing loops better In-Reply-To: References: Message-ID: <20210607000318.4d36b9cb@natsu> On Sun, 6 Jun 2021 11:13:36 +0200 "Jason A. Donenfeld" wrote: > Specifically the change would be to not allow IP fragmentation of the > encrypted UDP packets. This way, in the case of a loop, eventually the > packet size exceeds MTU, and it gets dropped: dumb and effective. > Depending on how this discussion goes, a compromise would be to not > allow fragmentation, but only for forwarded and kernel-generated > packets, not not for locally generated userspace packets. That's more > complex and I don't like it as much as just disallowing IP > fragmentation all together. > > Pros: > - It solves the routing loop problem very simply. Doesn't TTL already solve this? > - Maybe people are running > wireguard-over-gre-over-vxlan-over-l2tp-over-pppoe-over-god-knows-what-else, > and this reduces the MTU to below 1280, yet they still want to put > IPv6 through wireguard, and are willing to accept the performance > implications. Not only that. Sometimes transparent bridging of 1500 MTU LANs is required. VXLAN does not allow tunnel endpoints to produce fragmented VXLAN packets. With WG we can fragment them one level lower, *and* gain a higher efficiency compared to hypothetical VXLAN's fragmentation, due to less header overhead on 2nd and further packets in a chain. It would be unfortunate if this will become no longer possible. It appears to me that people who might need to transparently join multiple Ethernet LANs due to legacy network topologies they have to work with, weird requirements, various legacy software etc, would outnumber those who even run WG over WG at all, let alone getting themselves into a routing loop that way. -- With respect, Roman From Jason at zx2c4.com Mon Jun 7 09:34:21 2021 From: Jason at zx2c4.com (Jason A. Donenfeld) Date: Mon, 7 Jun 2021 11:34:21 +0200 Subject: potentially disallowing IP fragmentation on wg packets, and handling routing loops better In-Reply-To: References: Message-ID: Hey folks, There seems to be a bit of confusion about *which* stage of fragmentation would be affected by the proposal, so I drew some diagrams to help illustrate what I'm talking about. Please take a look: https://data.zx2c4.com/potential-wg-fragmentation-proposal.png 1) Ingress fragmentation would not be affected by this and is not relevant for this discussion. This is the case in which a computer gets a packet for forwarding out of the wireguard interface, and it's larger than the interface's mtu, so the computer fragments it before passing it onto that interface. I'm not suggesting any change in this behavior. 2) Local egress fragmentation WOULD be affected by this and is the most relevant thing in this discussion. In this case, a packet that gets encrypted and winds up being larger than the mtu of the interface that the encrypted packet will go out of gets fragmented. In this case, we could likely respond with an ICMP packet or similar in-path error. But keep in mind this whole situation is local: it usually will only happen out of misconfiguration. The best fix for the diagram I drew would be for the administrator to decrease the MTU of the wireguard interface to 1412. 3) Path egress fragmentation COULD be affected by this, but doesn't have to be. In this case, we simply set "don't fragment" on encrypted egress packets, which means they won't be fragmented by other computers along the path. So, of those concerned about this, which concerns are actually about (2) and (3)? Of those, which ones are about (2)? If you have concerns specifically about (2) that couldn't be fixed with reasonable system administration, I'd like to hear why and what the setup is that leads to that situation. As an aside, Roman asked about TTL. When tunneling, the outer packet header always must take the new TTL of the route to the tunnel endpoint, and not do anything with the potentially much smaller inner TTL. So with tunneling, you can't quite rely on the TTL to drop to zero as you'd wish. Hence, I'm interested in using the natural packet size expansion instead. Thanks for the discussion so far. I'm very interested to read clarifying points about applicability to case (2) (and to a lesser extent, about case (3)). Thanks, Jason From rcmcdonald91 at gmail.com Mon Jun 7 11:05:43 2021 From: rcmcdonald91 at gmail.com (Christian McDonald) Date: Mon, 7 Jun 2021 07:05:43 -0400 Subject: Certain private keys being mangled by wg on FreeBSD In-Reply-To: References: Message-ID: Ah that makes sense. I spent some quality time playing with the bit arithmetic and I see what you mean now. Thanks for that snippet and direction. One byproduct of this exercise was some code that I whipped up that can at least detect a clamped vs unclamped key. This might prove useful for informing a user of what is going on and thus eliminating this class of erroneous bug report entirely. I'm really not sure hiding the private keys entirely in the UI is the right thing to do, especially if it seems that key generators should really be pre-clamping keys (thanks for reaching out to Mullvad btw) and not dishing out unclamped keys to begin with. On Sun, Jun 6, 2021 at 12:21 PM Jason A. Donenfeld wrote: > > On 6/6/21, Christian McDonald wrote: > > Would it not be better for wg to just fail outright instead of > > transforming a poorly generated key entered by a user, regardless of > > where the key came from? Especially if that problematic key passes the > > regex validation that was provided in another thread in this email > > list? > > No, it would not be better. There is nothing wrong with using those > keys. They're not "poorly generated" or "problematic" or dangerous in > the least. This is only a concern with your UI. > > The kernel is doing the correct thing -- clamping keys -- and > displaying an unambiguous identifier to the user: the key that it will > actually be using. > > I suspect the best thing to do for your UI would be to hide private > (and preshared) keys, and only show public keys, unless explicitly > exported into a config file. This not only reduces potential confusion > with this issue, but mitigates another potential footgun down the > line. It's also what wg(8)'s show command does by default (while > showconf will export all). -- R. Christian McDonald M: (616) 856-9291 E: rcmcdonald91 at gmail.com From rm at romanrm.net Mon Jun 7 11:13:13 2021 From: rm at romanrm.net (Roman Mamedov) Date: Mon, 7 Jun 2021 16:13:13 +0500 Subject: potentially disallowing IP fragmentation on wg packets, and handling routing loops better In-Reply-To: References: Message-ID: <20210607161313.764eb5d6@natsu> On Mon, 7 Jun 2021 11:34:21 +0200 "Jason A. Donenfeld" wrote: > 2) Local egress fragmentation WOULD be affected by this and is the > most relevant thing in this discussion. In this case, a packet that > gets encrypted and winds up being larger than the mtu of the interface > that the encrypted packet will go out of gets fragmented. In this > case, we could likely respond with an ICMP packet or similar in-path > error. But keep in mind this whole situation is local: it usually will > only happen out of misconfiguration. The best fix for the diagram I > drew would be for the administrator to decrease the MTU of the > wireguard interface to 1412. In the L2 tunneling scenario the large VXLAN packets are generated locally, as it will be common for the same host (aka "the router") to be both a WG peer and a VXLAN VTEP, so it is going to be affected. > So, of those concerned about this, which concerns are actually about > (2) and (3)? Of those, which ones are about (2)? If you have concerns > specifically about (2) that couldn't be fixed with reasonable system > administration, I'd like to hear why and what the setup is that leads > to that situation. My described case is being able to transparently bridge two Ethernet LANs. Hopefully the answer isn't "you don't really need to do that" or "apply reasonable system administration and set up routing instead". > As an aside, Roman asked about TTL. When tunneling, the outer packet > header always must take the new TTL of the route to the tunnel > endpoint, and not do anything with the potentially much smaller inner > TTL. As far as I can see the inner TTL is not smaller than usual on WG tunnels (64). You could inherit it to the outside of the tunnel, like GRE does: https://serverfault.com/questions/827239/gre-tunnel-ttl-number But of course that's leaking a tiny bit of information about the encrypted tunnel, dunno how critical that would be. -- With respect, Roman From nico.schottelius at ungleich.ch Mon Jun 7 11:18:35 2021 From: nico.schottelius at ungleich.ch (Nico Schottelius) Date: Mon, 07 Jun 2021 13:18:35 +0200 Subject: potentially disallowing IP fragmentation on wg packets, and handling routing loops better In-Reply-To: References: Message-ID: <87fsxtvs6s.fsf@ungleich.ch> Hey Jason, Jason A. Donenfeld writes: > Hey folks, > > There seems to be a bit of confusion about *which* stage of > fragmentation would be affected by the proposal, so I drew some > diagrams to help illustrate what I'm talking about. Please take a > look: > > https://data.zx2c4.com/potential-wg-fragmentation-proposal.png I love the math: 2792 = 1420 + 1420 = 1500 + 1500 Joke aside, ... > 1) Ingress fragmentation would not be affected by this and is not > relevant for this discussion. This is the case in which a computer > gets a packet for forwarding out of the wireguard interface, and it's > larger than the interface's mtu, so the computer fragments it before > passing it onto that interface. I'm not suggesting any change in this > behavior. I believe this is something wireguard cannot influence *anyway* as the sending side can send any sized packet towards us. > 2) Local egress fragmentation WOULD be affected by this and is the > most relevant thing in this discussion. In this case, a packet that > gets encrypted and winds up being larger than the mtu of the interface > that the encrypted packet will go out of gets fragmented. In this > case, we could likely respond with an ICMP packet or similar in-path > error. But keep in mind this whole situation is local: it usually will > only happen out of misconfiguration. The best fix for the diagram I > drew would be for the administrator to decrease the MTU of the > wireguard interface to 1412. So how does that behave in the situation that the upstream interface or routes change? Let's say WG MTU = 1412, original PMTU = 1500, decreases to 1420. Would that reduce the WG mtu automatically to 1332? I guess not. So what happens with packets arrive with size = 1420? > 3) Path egress fragmentation COULD be affected by this, but doesn't > have to be. In this case, we simply set "don't fragment" on encrypted > egress packets, which means they won't be fragmented by other > computers along the path. That's true, but then it would be required to fragment them locally, wouldn't it? I'm trying to wrap my head around this in comparison to IPv6/IPv4: In the IPv6 world we don't have fragmentation on the path, it's always client based. In the IPv4 world routers can dis/re-assemble packets on the way. If I understand it correctly, you are somewhat suggestion that wireguard behaves a bit like an IPv6 router, albeit for both the v6 and the v4 world. Is that comparison making sense somehow? I think it would be easier to understand, if there was a demo case, a sample tunnel that rejects packets, if fragmentation is needed. What would be the appropriate ICMP message for an IPv4 packet that does not include the DF bit? So far, I'm not fully convinced the approach is a smart way, especially not when it comes to handling network debugging and given that we do already have a TTL that should be a loop prevention as well. Best regards, Nico -- Sustainable and modern Infrastructures by ungleich.ch From Jason at zx2c4.com Mon Jun 7 11:27:10 2021 From: Jason at zx2c4.com (Jason A. Donenfeld) Date: Mon, 7 Jun 2021 13:27:10 +0200 Subject: potentially disallowing IP fragmentation on wg packets, and handling routing loops better In-Reply-To: <20210607161313.764eb5d6@natsu> References: <20210607161313.764eb5d6@natsu> Message-ID: Hi Roman, On Mon, Jun 7, 2021 at 1:13 PM Roman Mamedov wrote: > In the L2 tunneling scenario the large VXLAN packets are generated locally, as > it will be common for the same host (aka "the router") to be both a WG peer > and a VXLAN VTEP, so it is going to be affected. Can you walk me through your use case a bit more, so I can wrap my mind around the requirements? ingress --plain--> wireguard --wireguard[plain]--> vxlan --vxlan[wireguard[plain]]--> egress So my question is, why can't you set wireguard's MTU to 80 bytes less than vxlan's MTU? What's preventing that or making it infeasible? Jason From rm at romanrm.net Mon Jun 7 11:46:17 2021 From: rm at romanrm.net (Roman Mamedov) Date: Mon, 7 Jun 2021 16:46:17 +0500 Subject: potentially disallowing IP fragmentation on wg packets, and handling routing loops better In-Reply-To: References: <20210607161313.764eb5d6@natsu> Message-ID: <20210607164617.6bf015d1@natsu> On Mon, 7 Jun 2021 13:27:10 +0200 "Jason A. Donenfeld" wrote: > Can you walk me through your use case a bit more, so I can wrap my mind > around the requirements? > > ingress --plain--> wireguard --wireguard[plain]--> vxlan --vxlan[wireguard[plain]]--> egress Not sure I understand your scheme correctly. In any case, the path of a packet would be... On peer 1: * plain Ethernet -> wrapped into VXLAN -> encrypted into WireGuard On peer 2: * decrypted from WireGuard -> unwrapped from VXLAN -> plain Ethernet > So my question is, why can't you set wireguard's MTU to 80 bytes less > than vxlan's MTU? What's preventing that or making it infeasible? To transparently bridge two Ethernet LANs, a VXLAN interface needs to join an L2 bridge. All interfaces that are members of a bridge must have the same MTU. As such, br0 members on both sides: eth0 (MTU 1500) vx0 (MTU 1500) VXLAN transports full L2 frames encapsulating them into UDP. To fit the full 1500-byte packet and accounting for VXLAN and related IP overheads, the resulting packet size is 1574 bytes. So this same host that just generated the 1574-byte encapsulated VXLAN packet with something it received via its eth0 port, now needs to send it further to its WG peer(s). For this to succeed, the in-tunnel WG MTU needs to be 1574 or more, not 1412 or 1420, as VXLAN itself can't be fragmented[1]; or even if it could, that would mean a much worse overhead ratio than currently. [1] https://datatracker.ietf.org/doc/html/rfc7348#section-4.3 -- With respect, Roman From peter at fiberdirekt.se Mon Jun 7 11:55:01 2021 From: peter at fiberdirekt.se (Peter Linder) Date: Mon, 7 Jun 2021 13:55:01 +0200 Subject: potentially disallowing IP fragmentation on wg packets, and handling routing loops better In-Reply-To: <20210607164617.6bf015d1@natsu> References: <20210607161313.764eb5d6@natsu> <20210607164617.6bf015d1@natsu> Message-ID: <43c72c2d-3649-b329-5ee5-78e891c69f04@fiberdirekt.se> This is indeed the case for me, spot on. On 2021-06-07 13:46, Roman Mamedov wrote: > So this same host that just generated the 1574-byte encapsulated VXLAN packet > with something it received via its eth0 port, now needs to send it further to > its WG peer(s). For this to succeed, the in-tunnel WG MTU needs to be 1574 or > more, not 1412 or 1420, as VXLAN itself can't be fragmented[1]; or even if it > could, that would mean a much worse overhead ratio than currently. From dfawcus+lists-wireguard at employees.org Mon Jun 7 11:58:29 2021 From: dfawcus+lists-wireguard at employees.org (Derek Fawcus) Date: Mon, 7 Jun 2021 12:58:29 +0100 Subject: potentially disallowing IP fragmentation on wg packets, and handling routing loops better In-Reply-To: References: Message-ID: On Sun, Jun 06, 2021 at 01:14:16PM +0200, Peter Linder wrote: > This would break things for me. We're doing a lot of L2 over L3 site to > site stuff and we are using wireguard as the outer layer. Inner layer is > vxlan or l2tpv3. > > In particular, people connect lots of stuff with no regard for MTU. For > some things it's also very hard to change so we just assume people > don't. Since the L3 network typically has the same MTU as the inner L2 > network, we need fragmentation. There is no practical way to be able to > tell hosts on the L2 network about the limited mtu, for all we know they > don't even run IP.... I've not looked in to vxlan much, but for L2TP you always have recourse to RFC 4623, where the MRU & MRRU can be exchanged. DF From Jason at zx2c4.com Mon Jun 7 12:52:13 2021 From: Jason at zx2c4.com (Jason A. Donenfeld) Date: Mon, 7 Jun 2021 14:52:13 +0200 Subject: Certain private keys being mangled by wg on FreeBSD In-Reply-To: References: Message-ID: On 6/7/21, Christian McDonald wrote: > One byproduct of this exercise was some code that I whipped > up that can at least detect a clamped vs unclamped key. This might > prove useful for informing a user of what is going on and thus > eliminating this class of erroneous bug report entirely. I'd recommend *not* introducing users to weird ideas like clamping or key transformation. While learning new concepts and bit masking in PHP is undoubtedly fun, those concerns shouldn't be user-facing. There's nothing wrong or dangerous about unclamped scalars passed to a proper 25519 implementation, because the implementation will clamp on input. Throwing an "X-vs-unX" distinction to users will just result in pointless fear mongering nonsense. Instead just communicate the identity of an interface by its public key, rather than its private key. If you're not willing to hide or mask private keys (which you really should), then at least deemphasize them? From jwh at zorins.us Sun Jun 6 22:33:54 2021 From: jwh at zorins.us (Joe Holden) Date: Mon, 7 Jun 2021 00:33:54 +0200 Subject: potentially disallowing IP fragmentation on wg packets, and handling routing loops better In-Reply-To: <20210607000318.4d36b9cb@natsu> References: <20210607000318.4d36b9cb@natsu> Message-ID: <0cac2d72-debd-bff6-2404-6b3aabbdeb0d@zorins.us> On 2021-06-06 21:03, Roman Mamedov wrote: > On Sun, 6 Jun 2021 11:13:36 +0200 > "Jason A. Donenfeld" wrote: > >> Specifically the change would be to not allow IP fragmentation of the >> encrypted UDP packets. This way, in the case of a loop, eventually the >> packet size exceeds MTU, and it gets dropped: dumb and effective. >> Depending on how this discussion goes, a compromise would be to not >> allow fragmentation, but only for forwarded and kernel-generated >> packets, not not for locally generated userspace packets. That's more >> complex and I don't like it as much as just disallowing IP >> fragmentation all together. >> >> Pros: >> - It solves the routing loop problem very simply. > > Doesn't TTL already solve this? > >> - Maybe people are running >> wireguard-over-gre-over-vxlan-over-l2tp-over-pppoe-over-god-knows-what-else, >> and this reduces the MTU to below 1280, yet they still want to put >> IPv6 through wireguard, and are willing to accept the performance >> implications. > > Not only that. Sometimes transparent bridging of 1500 MTU LANs is required. > > VXLAN does not allow tunnel endpoints to produce fragmented VXLAN packets. > > With WG we can fragment them one level lower, *and* gain a higher efficiency > compared to hypothetical VXLAN's fragmentation, due to less header overhead on > 2nd and further packets in a chain. > > It would be unfortunate if this will become no longer possible. > > It appears to me that people who might need to transparently join multiple > Ethernet LANs due to legacy network topologies they have to work with, weird > requirements, various legacy software etc, would outnumber those who even run > WG over WG at all, let alone getting themselves into a routing loop that way. > All of the above, really - not allowing "full" sized frames over WG breaks a huge number of use cases - even simple ones, because regardless of how much it's wished to be true, in reality pmtu isn't very useful and doesn't work for many cases even in an environment where it isn't completely broken by firewalls/misconfiguration. A [probably common] simple example is where there are 1500 byte speakers on either side of a WG link (e.g. the internet, or some satellite site) - having a <1500 byte link in the middle will break many applications, in particular especially UDP based protocols. Unfortunately the better solution is likely to make it configurable, or allow fragmentation for forwarded traffic (since the host already knows the mtu, this solves the problem without requiring any user config) - although understandably you don't want to add more complexity thanks From rm at romanrm.net Mon Jun 7 18:50:34 2021 From: rm at romanrm.net (Roman Mamedov) Date: Mon, 7 Jun 2021 23:50:34 +0500 Subject: potentially disallowing IP fragmentation on wg packets, and handling routing loops better In-Reply-To: <20210607164617.6bf015d1@natsu> References: <20210607161313.764eb5d6@natsu> <20210607164617.6bf015d1@natsu> Message-ID: <20210607235034.024e6c6b@natsu> On Mon, 7 Jun 2021 16:46:17 +0500 Roman Mamedov wrote: > On Mon, 7 Jun 2021 13:27:10 +0200 > "Jason A. Donenfeld" wrote: > > > Can you walk me through your use case a bit more, so I can wrap my mind > > around the requirements? > > > > ingress --plain--> wireguard --wireguard[plain]--> vxlan --vxlan[wireguard[plain]]--> egress > > Not sure I understand your scheme correctly. In any case, the path of a > packet would be... > > On peer 1: > > * plain Ethernet -> wrapped into VXLAN -> encrypted into WireGuard > > On peer 2: > > * decrypted from WireGuard -> unwrapped from VXLAN -> plain Ethernet > > > So my question is, why can't you set wireguard's MTU to 80 bytes less > > than vxlan's MTU? What's preventing that or making it infeasible? > > To transparently bridge two Ethernet LANs, a VXLAN interface needs to join an > L2 bridge. All interfaces that are members of a bridge must have the same MTU. > > As such, br0 members on both sides: > eth0 (MTU 1500) > vx0 (MTU 1500) > > VXLAN transports full L2 frames encapsulating them into UDP. To fit the > full 1500-byte packet and accounting for VXLAN and related IP overheads, > the resulting packet size is 1574 bytes. > > So this same host that just generated the 1574-byte encapsulated VXLAN packet > with something it received via its eth0 port, now needs to send it further to > its WG peer(s). For this to succeed, the in-tunnel WG MTU needs to be 1574 or > more, not 1412 or 1420, as VXLAN itself can't be fragmented[1]; or even if it > could, that would mean a much worse overhead ratio than currently. > > [1] https://datatracker.ietf.org/doc/html/rfc7348#section-4.3 In case you are not convinced by this case, would you consider at least allowing fragmentation when WG's in-tunnel MTU is set to >=1500? Because this is the user effectively saying "yes I know this is not gonna fit in one packet, I want to rely on WG packets being fragmented", but without the need for extra knobs. -- With respect, Roman From huzhengmian at gmail.com Mon Jun 7 16:02:16 2021 From: huzhengmian at gmail.com (zhengmian hu) Date: Mon, 7 Jun 2021 12:02:16 -0400 Subject: Cannot use localhost as endpoint for wireguard-windows Message-ID: I might encounter the same problem as https://lists.zx2c4.com/pipermail/wireguard/2020-August/005801.html When endpoint is set as 127.0.0.1:51820, the log shows like follows: [TUN] [xxx] peer(xxx) Received handshake response [TUN] [xxx] peer(xxx) - Retrying handshake because we stopped hearing back after 15 seconds. [TUN] [xxx] peer(xxx) - Sending handshake initiation [TUN] [xxx] peer(xxx) - Handshake did not complete after 5 seconds, retrying (try 2) [TUN] [xxx] peer(xxx) - Sending handshake initiation [TUN] [xxx] peer(xxx) - Handshake did not complete after 5 seconds, retrying (try 4) ... In Wireshark, I can see that only the first handshake initiation package is sent. There are no following packages even if the log shows "Sending handshake initiation". However, when I use another ip of the local computer for the endpoint, e.g. 192.168.1.2, everything works fine. Is it a known bug of wireguard-windows? Is there any existing formal description on the scope of this bug? Is there any way to bypass this problem? From tigger2014g at gmail.com Mon Jun 7 19:17:58 2021 From: tigger2014g at gmail.com (ben edmunds) Date: Mon, 7 Jun 2021 20:17:58 +0100 Subject: Certain private keys being mangled by wg on FreeBSD In-Reply-To: References: Message-ID: The issue here for pfSense is that the private key will be viewable just like it is within native wireguard clients in the peer config options and needs to be viewable here for admin and debug purposes. With regards to clamping and hiding this from users its tricky as it leads to red heroin issues as people debug the tunnels via showcase for example and will see a different key to which they entered in the UI. So the only logical option is to: 1) inform the admin that the key has been clamped 2) show the admin the clamped key which they can see whilst debugging. By not showing this to the user to avoid confusion we actually would create confusion in this scenario as the kernel module is performing the clamping but the user would have no knowledge of this and leads to issues being opened that are a non issue. The aim is not to show the users anything about clamping unless the key needs to be clamped as it was not clamped already. I belive it is key to remember that pfSense is not an end user application/tool and designed to be used by admins & network engineers so should be considered power users who are capable of being exposed to more information. Regards Tigger2014 From Jason at zx2c4.com Tue Jun 8 13:20:32 2021 From: Jason at zx2c4.com (Jason A. Donenfeld) Date: Tue, 8 Jun 2021 15:20:32 +0200 Subject: Certain private keys being mangled by wg on FreeBSD In-Reply-To: References: Message-ID: On Tue, Jun 8, 2021 at 1:00 PM ben edmunds wrote: > By not showing this to the user to avoid confusion we actually would > create confusion in this scenario as the kernel module is performing the > clamping but the user would have no knowledge of this and leads to > issues being opened that are a non issue. The aim is not to show the > users anything about clamping unless the key needs to be clamped as it > was not clamped already. I think you are making a mistake, and introducing users to the concept of clamping will just breed confusion. > I belive it is key to remember that pfSense is not an end user > application/tool and designed to be used by admins & network engineers You made that same point on some Github bug report; it's not news to me, and it still doesn't change my point of view. Introducing excessive complexity and superfluous technical information results in problematic decisions, configurations, and decision calculuses, even for the most powerful of power users. In particular, here, I think it's only going to sow confusion and bad information to expose users to contingent details about "valid private keys" and "less valid private keys" with weird nerdy language like "unclamped". Because the fact is, any 256-bit bitstring generated from a csprng is a fine private key. From frederic.deghetto at free.fr Tue Jun 8 19:11:06 2021 From: frederic.deghetto at free.fr (=?utf-8?Q?Fr=C3=A9d=C3=A9ric_Deghetto?=) Date: Tue, 8 Jun 2021 21:11:06 +0200 Subject: Wireguard active but no routing to the remote host Message-ID: Hi, I have two raspberry's at different locations with wireguard on it (installation carried out thanks to pivpn). If I use wireguard to be connected to the remote raspberry from my macbook or iPad, the connection if activated but I am still on my local network. If I do the same with my android phone : I am on the network where the remote raspberry is. On my MacBook/iPad, If I stop the wireless connection to my internet provider (which is the same than the one of the remote raspberry), and connect trough my phone by access point, then it works. Any idea how I can troubleshoot this ? Many thanks, Frederic From diggest at gmail.com Wed Jun 9 23:26:49 2021 From: diggest at gmail.com (Vasili Pupkin) Date: Thu, 10 Jun 2021 02:26:49 +0300 Subject: potentially disallowing IP fragmentation on wg packets, and handling routing loops better In-Reply-To: References: Message-ID: Hi, Talking about my example with chained VPNs. It is a misconfiguration but not intentional, and no responsible administrator can solve this because client really has no way to tell a VPN provider what MTU he needs. Technically VP providers can make such interface for clients but none of four VPN providers I've tried during the last three years has this. Usually VPN providers just use a default ?wg-quick? MTU in their server configs. On the client side it is (2 local egress fragmentation), client knows both of the MTU values, but on the server side it already needs (3 path egress fragmentation). If I understand the terminology correctly. For a packet on the route: remote host -> VPN provider 1 -> VPN provider 2 -> client. An unecrypted packet comes form remote host, the VPN provider 1 just sends a bigger encrypted packet to the VPN provider 2 and even if it responds with ICMP Fragmentation Needed to the VPN provider 1, it will be ignored and would not be repeated in the unecrypted channel to remote host. So the remote host will never know why the packet was dropped and it will slow down PMTUD. How difficult it is and what security implications it will have if WireGuard do capture ICMP Fragmentation Needed responses and repeat them in unencrypted channel? On 07.06.2021 12:34, Jason A. Donenfeld wrote: > Hey folks, > > There seems to be a bit of confusion about *which* stage of > fragmentation would be affected by the proposal, so I drew some > diagrams to help illustrate what I'm talking about. Please take a > look: > > https://data.zx2c4.com/potential-wg-fragmentation-proposal.png > > 1) Ingress fragmentation would not be affected by this and is not > relevant for this discussion. This is the case in which a computer > gets a packet for forwarding out of the wireguard interface, and it's > larger than the interface's mtu, so the computer fragments it before > passing it onto that interface. I'm not suggesting any change in this > behavior. > > 2) Local egress fragmentation WOULD be affected by this and is the > most relevant thing in this discussion. In this case, a packet that > gets encrypted and winds up being larger than the mtu of the interface > that the encrypted packet will go out of gets fragmented. In this > case, we could likely respond with an ICMP packet or similar in-path > error. But keep in mind this whole situation is local: it usually will > only happen out of misconfiguration. The best fix for the diagram I > drew would be for the administrator to decrease the MTU of the > wireguard interface to 1412. > > 3) Path egress fragmentation COULD be affected by this, but doesn't > have to be. In this case, we simply set "don't fragment" on encrypted > egress packets, which means they won't be fragmented by other > computers along the path. > > So, of those concerned about this, which concerns are actually about > (2) and (3)? Of those, which ones are about (2)? If you have concerns > specifically about (2) that couldn't be fixed with reasonable system > administration, I'd like to hear why and what the setup is that leads > to that situation. > > As an aside, Roman asked about TTL. When tunneling, the outer packet > header always must take the new TTL of the route to the tunnel > endpoint, and not do anything with the potentially much smaller inner > TTL. So with tunneling, you can't quite rely on the TTL to drop to > zero as you'd wish. Hence, I'm interested in using the natural packet > size expansion instead. > > Thanks for the discussion so far. I'm very interested to read > clarifying points about applicability to case (2) (and to a lesser > extent, about case (3)). > > Thanks, > Jason > From and at mullvad.net Thu Jun 10 11:44:45 2021 From: and at mullvad.net (Andrej Mihajlov) Date: Thu, 10 Jun 2021 13:44:45 +0200 Subject: WireGuard is broken on iOS 15 beta Message-ID: <760B19F6-17E7-4276-B2E2-DFE07AB19323@mullvad.net> Hi, Just though to give you heads up. iOS 15 beta is out and it looks like the old way of obtaining the tunnel device descriptor is no longer working as the internal structure of NEPacketFlow has changed. It?s been a ticking bomb form the start as with any private API that is subject to change without a warning. In the short term, researching what Apple changed and looking for a quick fix for that. Longer term I believe we should have a better integration between wireguard-go and Network extension on iOS. Cheers, Andrej Mihajlov From Jason at zx2c4.com Thu Jun 10 11:51:47 2021 From: Jason at zx2c4.com (Jason A. Donenfeld) Date: Thu, 10 Jun 2021 13:51:47 +0200 Subject: WireGuard is broken on iOS 15 beta In-Reply-To: <760B19F6-17E7-4276-B2E2-DFE07AB19323@mullvad.net> References: <760B19F6-17E7-4276-B2E2-DFE07AB19323@mullvad.net> Message-ID: Patches with a proper performance comparison of the two techniques in the description would be appreciated. Thanks, Jason From Jason at zx2c4.com Thu Jun 10 15:39:20 2021 From: Jason at zx2c4.com (Jason A. Donenfeld) Date: Thu, 10 Jun 2021 17:39:20 +0200 Subject: WireGuard is broken on iOS 15 beta In-Reply-To: References: <760B19F6-17E7-4276-B2E2-DFE07AB19323@mullvad.net> Message-ID: I won't have an updated OS to test this out in until next week at the earliest, but perhaps this hack will work? https://git.zx2c4.com/wireguard-apple/commit/?h=jd/fd-search-hack Let me know if that is successful, or if it blows up. Jason From crawshaw at tailscale.com Thu Jun 10 22:33:53 2021 From: crawshaw at tailscale.com (David Crawshaw) Date: Thu, 10 Jun 2021 15:33:53 -0700 Subject: WireGuard is broken on iOS 15 beta In-Reply-To: References: <760B19F6-17E7-4276-B2E2-DFE07AB19323@mullvad.net> Message-ID: I finally managed to get a device running iPadOS 15 and an Xcode 13 beta. This technique works. Some modified code with logging: func tunnelFD(provider: NEPacketTunnelProvider) -> Int32 { os_log("tunnelFD searching") var buf = [CChar](repeating: 0, count: Int(IFNAMSIZ)) for fd: Int32 in 4...64 { var len = socklen_t(buf.count) if getsockopt(fd, 2 /* SYSPROTO_CONTROL */, 2, &buf, &len) == 0 { let str = String(cString: buf) os_log("%{public}@", "tunnelFD \(fd): \(str)") if str.starts(with: "utun") { os_log("tunnelFD found likely fd") return fd } } } os_log("tunnelFD found nothing") return -1 } produces the logs: tunnelFD searching tunnelFD 7: utun3 tunnelFD found likely fd Note that the current technique of calling provider.packetFlow.value(forKeyPath: "socket.fileDescriptor") causes the NetworkExtension to crash under iOS 15. The logs say the kernel sandbox killed it: Sandbox: process(525) deny(2) file-test-existence /private/etc/.mdns_debug On Thu, Jun 10, 2021 at 8:44 AM Jason A. Donenfeld wrote: > > I won't have an updated OS to test this out in until next week at the > earliest, but perhaps this hack will work? > https://git.zx2c4.com/wireguard-apple/commit/?h=jd/fd-search-hack > Let me know if that is successful, or if it blows up. > > Jason From enn00th at gmail.com Thu Jun 10 16:20:51 2021 From: enn00th at gmail.com (Alex Sivchev) Date: Thu, 10 Jun 2021 19:20:51 +0300 Subject: MSIs and WDAC Message-ID: Hi, The MSIs' customactions.dll is unsigned and therefore only Hash or FileName rules [1] can be used to allow it in WDAC policies. Can it be signed like the rest so that more reasonable and update-proof policies could be created? A patch is attached. Regards, Alex [1] https://docs.microsoft.com/en-us/windows/security/threat-protection/windows-defender-application-control/select-types-of-rules-to-create#windows-defender-application-control-file-rule-levels -------------- next part -------------- A non-text attachment was scrubbed... Name: installer-build-sign.patch Type: text/x-patch Size: 1592 bytes Desc: not available URL: From dimitar.vassilev at gmail.com Fri Jun 11 06:23:40 2021 From: dimitar.vassilev at gmail.com (Dimitar Vassilev) Date: Fri, 11 Jun 2021 09:23:40 +0300 Subject: openwrt interface does not have a public key after upgrade from librecmc 1.4.8 to openwrt 21.02-rc2 to openwrt 21.02-rc2 ath79 Message-ID: Greetings, I upgraded an WNDR3800 from librecmc 1.4.8/ath71 to openwrt 21.02-rc2 ath79 and reapplied the librecmc 1.4.8 configuration. I'm seeing now the interface configuration in the CLI and in Luci with all the keys, but when I click on Status->Wireguard I get a message interface does not have a public key. I did have a look on https://forum.openwrt.org/t/wireguard-interface-does-not-have-a-public-key/37549 and the other threads, but even when after applying the work-arounds for DNS rebind protection still the same. My Luci is using self-signed certs as well. Any clues where to look further? I will be testing the device today to see how it really works. I do have some overlapping IP ranges in the configs that make me reluctant to test how site A machine works as a client of site B, while the actual site A is running remotely over the VPN. Thanks, Dimitar From and at mullvad.net Fri Jun 11 07:16:25 2021 From: and at mullvad.net (Andrej Mihajlov) Date: Fri, 11 Jun 2021 09:16:25 +0200 Subject: WireGuard is broken on iOS 15 beta In-Reply-To: References: <760B19F6-17E7-4276-B2E2-DFE07AB19323@mullvad.net> Message-ID: IIRC one thing to consider with that lookup: iOS enables users to run Personal VPN and Custom VPN (aka WireGuard) side-by-side so there is a chance you may pick the wrong utun. > On 11 Jun 2021, at 00:33, David Crawshaw wrote: > > I finally managed to get a device running iPadOS 15 and an Xcode 13 > beta. This technique works. Some modified code with logging: > > func tunnelFD(provider: NEPacketTunnelProvider) -> Int32 { > os_log("tunnelFD searching") > var buf = [CChar](repeating: 0, count: Int(IFNAMSIZ)) > for fd: Int32 in 4...64 { > var len = socklen_t(buf.count) > if getsockopt(fd, 2 /* SYSPROTO_CONTROL */, 2, &buf, &len) == 0 { > let str = String(cString: buf) > os_log("%{public}@", "tunnelFD \(fd): \(str)") > if str.starts(with: "utun") { > os_log("tunnelFD found likely fd") > return fd > } > } > } > os_log("tunnelFD found nothing") > return -1 > } > > produces the logs: > > tunnelFD searching > tunnelFD 7: utun3 > tunnelFD found likely fd > > Note that the current technique of calling > provider.packetFlow.value(forKeyPath: "socket.fileDescriptor") causes > the NetworkExtension to crash under iOS 15. The logs say the kernel > sandbox killed it: > > Sandbox: process(525) deny(2) file-test-existence /private/etc/.mdns_debug > > On Thu, Jun 10, 2021 at 8:44 AM Jason A. Donenfeld wrote: >> >> I won't have an updated OS to test this out in until next week at the >> earliest, but perhaps this hack will work? >> https://git.zx2c4.com/wireguard-apple/commit/?h=jd/fd-search-hack >> Let me know if that is successful, or if it blows up. >> >> Jason From Jason at zx2c4.com Fri Jun 11 07:53:59 2021 From: Jason at zx2c4.com (Jason A. Donenfeld) Date: Fri, 11 Jun 2021 09:53:59 +0200 Subject: WireGuard is broken on iOS 15 beta In-Reply-To: References: <760B19F6-17E7-4276-B2E2-DFE07AB19323@mullvad.net> Message-ID: On 6/11/21, Andrej Mihajlov wrote: > IIRC one thing to consider with that lookup: iOS enables users to run > Personal VPN and Custom VPN (aka WireGuard) side-by-side so there is a > chance you may pick the wrong utun. That doesn't make any sense. File descriptors are not OS-global; they're process-local. That's how Unix FDs have worked since forever. Unless you're suggesting "personal VPN" is somehow resident in the same network extension process as WireGuard's "custom VPN"? By the way, did the experiment in your branch work? I'd prefer a direct route to brute forcing FDs, if possible. But if not, seems like my kludge might ultimately do the trick. From and at mullvad.net Fri Jun 11 08:42:14 2021 From: and at mullvad.net (Andrej Mihajlov) Date: Fri, 11 Jun 2021 10:42:14 +0200 Subject: WireGuard is broken on iOS 15 beta In-Reply-To: References: <760B19F6-17E7-4276-B2E2-DFE07AB19323@mullvad.net> Message-ID: <3BD85DF3-A8B2-45BF-A78C-D03524A226DC@mullvad.net> Jason, My bad, you?re right regarding the Personal VPN. I just ran your patch and it works great. As David pointed out, the call to valueForKeyPath should be guarded because it throws exception if the given key path does not exist. I use the availability check to bruteforce utun on iOS 15, macOS 12 and onwards while keeping the Key value coding approach on older iOS and macOS as we know that it works great on iOS < 15 and macOS < 12. Just ran the app on macOS 11.4 and it?s still working and using the old code path. I took a liberty to refactor the proposed solution (see: https://git.zx2c4.com/wireguard-apple/commit/?id=a7ccc8e3031f3502ea4b53a914d37589186e40f8) Cheers, Andrej > On 11 Jun 2021, at 09:53, Jason A. Donenfeld wrote: > > On 6/11/21, Andrej Mihajlov wrote: >> IIRC one thing to consider with that lookup: iOS enables users to run >> Personal VPN and Custom VPN (aka WireGuard) side-by-side so there is a >> chance you may pick the wrong utun. > > That doesn't make any sense. File descriptors are not OS-global; > they're process-local. That's how Unix FDs have worked since forever. > Unless you're suggesting "personal VPN" is somehow resident in the > same network extension process as WireGuard's "custom VPN"? > > By the way, did the experiment in your branch work? I'd prefer a > direct route to brute forcing FDs, if possible. But if not, seems like > my kludge might ultimately do the trick. From Jason at zx2c4.com Fri Jun 11 08:50:57 2021 From: Jason at zx2c4.com (Jason A. Donenfeld) Date: Fri, 11 Jun 2021 10:50:57 +0200 Subject: WireGuard is broken on iOS 15 beta In-Reply-To: <3BD85DF3-A8B2-45BF-A78C-D03524A226DC@mullvad.net> References: <760B19F6-17E7-4276-B2E2-DFE07AB19323@mullvad.net> <3BD85DF3-A8B2-45BF-A78C-D03524A226DC@mullvad.net> Message-ID: Can I infer from that that your technique didn't work? Some confirmation one way or another there would be useful, so I know whether to dedicate some RE time next week to it or not. If I can avoid the fd iteration, I'd prefer that. Jason From and at mullvad.net Fri Jun 11 09:02:40 2021 From: and at mullvad.net (Andrej Mihajlov) Date: Fri, 11 Jun 2021 11:02:40 +0200 Subject: WireGuard is broken on iOS 15 beta In-Reply-To: References: <760B19F6-17E7-4276-B2E2-DFE07AB19323@mullvad.net> <3BD85DF3-A8B2-45BF-A78C-D03524A226DC@mullvad.net> Message-ID: <8A754E6A-6D25-427E-AF60-4AF9A61DF1B7@mullvad.net> My approach worked too. However it relies on the private interfaces of NetworkExtension framework, which we?d probably want to avoid. In short it can be expressed in one line as: let fd = packetFlow.perform(Selector("interface")).toOpaque().advanced(by: 0x128).assumingMemoryBound(to: Int32.self).pointee But I bet that walking over fds is much safer. Cheers, Andrej > On 11 Jun 2021, at 10:50, Jason A. Donenfeld wrote: > > Can I infer from that that your technique didn't work? Some > confirmation one way or another there would be useful, so I know > whether to dedicate some RE time next week to it or not. If I can > avoid the fd iteration, I'd prefer that. > > Jason From Jason at zx2c4.com Fri Jun 11 09:09:18 2021 From: Jason at zx2c4.com (Jason A. Donenfeld) Date: Fri, 11 Jun 2021 11:09:18 +0200 Subject: WireGuard is broken on iOS 15 beta In-Reply-To: <8A754E6A-6D25-427E-AF60-4AF9A61DF1B7@mullvad.net> References: <760B19F6-17E7-4276-B2E2-DFE07AB19323@mullvad.net> <3BD85DF3-A8B2-45BF-A78C-D03524A226DC@mullvad.net> <8A754E6A-6D25-427E-AF60-4AF9A61DF1B7@mullvad.net> Message-ID: On 6/11/21, Andrej Mihajlov wrote: > let fd = packetFlow.perform(Selector("interface")).toOpaque().advanced(by: > 0x128).assumingMemoryBound(to: Int32.self).pointee Ahh, so that's some sort of 296 byte struct offset, rather than a symbolic name that fails gracefully. Ooof. Okay I'll poke around for other methods, I suppose, as that indeed seems potentially dangerous. Thanks nonetheless for the trick - helpful for getting started. Jason From wolletd at posteo.de Sat Jun 12 10:57:38 2021 From: wolletd at posteo.de (Eicke Herbertz) Date: Sat, 12 Jun 2021 10:57:38 +0000 Subject: [wg-setup] my scripts to manage WireGuard networks Message-ID: Hi folks, I want to share with you my `wg-setup` project that I use to manage my WireGuard servers and also setup Linux clients. These are CLI tools written in Bash (with some awk, of course), which already is a dependency of wg-quick. They are also aware of systemd-networkd and will use it instead of wg-quick, if it's enabled. It's main point is to keep the interface and config files in sync and to enforce unique hostnames, which are embedded into comments and don't depend on external files. I use this to generate DNS zones. There's no additional transmission involved. The client setup script generates a local key and prints a command that has to be executed on the server manually or by any automation of choice. Check out the setup example in the Readme: https://github.com/WolleTD/wg-setup --- Eicke Herbertz From dimitar.vassilev at gmail.com Mon Jun 14 07:51:19 2021 From: dimitar.vassilev at gmail.com (Dimitar Vassilev) Date: Mon, 14 Jun 2021 10:51:19 +0300 Subject: openwrt interface does not have a public key after upgrade from librecmc 1.4.8 to openwrt 21.02-rc2 to openwrt 21.02-rc2 ath79 In-Reply-To: References: Message-ID: ?? ??, 11.06.2021 ?. ? 9:23 ?. Dimitar Vassilev ??????: > > Greetings, > I upgraded an WNDR3800 from librecmc 1.4.8/ath71 to openwrt 21.02-rc2 > ath79 and reapplied the librecmc 1.4.8 configuration. I'm seeing now > the interface configuration in the CLI and in Luci with all the keys, > but when I click on Status->Wireguard I get a message interface does > not have a public key. > I did have a look on > https://forum.openwrt.org/t/wireguard-interface-does-not-have-a-public-key/37549 > and the other threads, but even when after applying the work-arounds > for DNS rebind protection still the same. My Luci is using self-signed > certs as well. > Any clues where to look further? I will be testing the device today to > see how it really works. I do have some overlapping IP ranges in the > configs that make me reluctant to test how site A machine works as a > client of site B, while the actual site A is running remotely over the > VPN. > Thanks, > Dimitar For the record - this is a cosmetics issue in my case. Once I put the uplink cable all came good. Disconnecting the VPN interface to avoid greater worries also helps [tm] Thanks, Dimitar From rcmcdonald91 at gmail.com Tue Jun 15 08:35:43 2021 From: rcmcdonald91 at gmail.com (Christian McDonald) Date: Tue, 15 Jun 2021 04:35:43 -0400 Subject: wg syncconf (and setconf) error when one or more endpoints is unresolvable Message-ID: Jason, Assume a tunnel with say 3 peers. Peer A is accessible via an IPv4 address, Peer B by some FQDN, and Peer C by some other FQDN. Let's also assume that Peer C was misconfigured with an unresolvable FQDN. wg syncconf (and setconf) fails with 'Name does not resolve...Configuration parsing error' Is it expected behavior in this case that *none* of the peer configurations are actually applied? It seems like a more appropriate behavior would be to go ahead and configure the remaining peers (Peer A + B) but only fail on the peer with an unresolvable endpoint (Peer C). It of course is completely possible to re-implement syncconf and setconf using explicit `wg set` calls as a workaround. Am I missing something here? Thanks, Christian -- R. Christian McDonald E: rcmcdonald91 at gmail.com From Jason at zx2c4.com Tue Jun 15 10:52:36 2021 From: Jason at zx2c4.com (Jason A. Donenfeld) Date: Tue, 15 Jun 2021 12:52:36 +0200 Subject: wg syncconf (and setconf) error when one or more endpoints is unresolvable In-Reply-To: References: Message-ID: This is intended behavior. DNS resolution happens at config parsing time. From lists at lonnie.abelbeck.com Tue Jun 15 13:23:59 2021 From: lists at lonnie.abelbeck.com (Lonnie Abelbeck) Date: Tue, 15 Jun 2021 08:23:59 -0500 Subject: wg syncconf (and setconf) error when one or more endpoints is unresolvable In-Reply-To: References: Message-ID: > On Jun 15, 2021, at 5:52 AM, Jason A. Donenfeld wrote: > > This is intended behavior. DNS resolution happens at config parsing time. Christian, While I appreciate Jason's strict DNS requirement, for the last 2.5 years our project has implemented a trivial patch [1] to ignore endpoint DNS failure. On a DNS failure, essentially ignoring the (optional) Endpoint= dns-hostname peer entry. This has worked well for our use case. WireGuard always starts. Lonnie [1] Ignore endpoint DNS failure https://github.com/astlinux-project/astlinux/blob/master/package/wireguard-tools/wireguard-tools-0001-ignore-endpoint-dns-failure.patch From gelstudios at gmail.com Mon Jun 14 20:03:26 2021 From: gelstudios at gmail.com (Eric Romano) Date: Mon, 14 Jun 2021 16:03:26 -0400 Subject: wintun interface sometimes can't start after wireguard-windows service dies Message-ID: Hey All, I ran into some unexpected behavior with a script that calls "/installtunnelservice" waits a bit, then calls "/uninstalltunnelservice". This happens in a loop with some connectivity checks in between. When the tunnel service itself ("wireguard.exe /tunnelservice ...") is killed, I noticed a log message that mentions "leftover residue" on the next service startup or next "/installtunnelservice" call. Most of the time, the client is still able to connect following the wintun residue message. snippet 1 > 2021-06-14 09:21:50.966: [TUN] [testClient] Starting WireGuard/0.3.14 (Windows 10.0.19042; amd64) 2021-06-14 09:21:50.966: [TUN] [testClient] Watching network interfaces 2021-06-14 09:21:50.970: [TUN] [testClient] Resolving DNS names 2021-06-14 09:21:50.970: [TUN] [testClient] Creating Wintun interface 2021-06-14 09:21:51.408: [TUN] [testClient] [Wintun] CreateAdapter: Creating adapter 2021-06-14 09:21:51.408: [TUN] [testClient] [Wintun] CreateAdapter: Requested GUID {01351A57-32EF-4644-C335-5A44C0808EC6} has leftover residue 2021-06-14 09:21:51.768: [TUN] [testClient] [Wintun] SelectDriver: Using existing driver 0.11 2021-06-14 09:21:53.679: [TUN] [testClient] Using Wintun/0.11 ... (normal behavior resumes) But on two different machines I've seen this error: "system cannot find the file specified", after which the tunnel device does not come up. At that point I attempted to uninstall the tunnel service, as well as uninstall+reinstall wireguard, with no success. A reboot was the only apparent way to recover from this state. snippet 2 > 2021-06-14 13:23:49.668: [TUN] [testClient] Starting WireGuard/0.3.14 (Windows 10.0.19042; amd64) 2021-06-14 13:23:49.677: [TUN] [testClient] Watching network interfaces 2021-06-14 13:23:49.686: [TUN] [testClient] Resolving DNS names 2021-06-14 13:23:49.686: [TUN] [testClient] Creating Wintun interface 2021-06-14 13:23:49.954: [TUN] [testClient] [Wintun] CreateAdapter: Creating adapter 2021-06-14 13:23:50.087: [TUN] [testClient] [Wintun] SelectDriver: Using existing driver 0.11 2021-06-14 13:23:50.939: [TUN] [testClient] [Wintun] CreateAdapter: Failed to setup adapter (code: 0x0, status: 0xc00002f0): The operation completed successfully. (Code 0x00000000) 2021-06-14 13:23:51.045: [TUN] [testClient] Unable to create Wintun interface: Error creating interface: The system cannot find the file specified. 2021-06-14 13:23:51.056: [TUN] [testClient] Shutting down ... dead from here onward Similar behavior with a different log message, unfortunately I don't know what was different about this test run. The same version of wireguard was in use. snippet 3 > 2021-06-10 18:42:22.044051: [TUN] [testClient] Starting WireGuard/0.3.14 (Windows 10.0.19042; amd64) 2021-06-10 18:42:22.044051: [TUN] [testClient] Watching network interfaces 2021-06-10 18:42:22.575650: [TUN] [testClient] Resolving DNS names 2021-06-10 18:42:22.575717: [TUN] [testClient] Creating Wintun interface 2021-06-10 18:42:27.920185: [TUN] [testClient] [Wintun] RegistryQuery: Querying registry value \REGISTRY\MACHINE\SYSTEM\ControlSet001\Control\Class\{4d36e972-e325-11ce-bfc1-08002be10318}\0011\NetCfgInstanceId failed: The system cannot find the file specified. (Code 0x00000002) 2021-06-10 18:42:27.920185: [TUN] [testClient] [Wintun] GetNetCfgInstanceIdFromHKEY: Failed to get \REGISTRY\MACHINE\SYSTEM\ControlSet001\Control\Class\{4d36e972-e325-11ce-bfc1-08002be10318}\0011\NetCfgInstanceId 2021-06-10 18:42:27.920185: [TUN] [testClient] [Wintun] GetNetCfgInstanceIdFromHKEY: Failed to get \REGISTRY\MACHINE\SYSTEM\ControlSet001\Control\Class\{4d36e972-e325-11ce-bfc1-08002be10318}\0011\NetCfgInstanceId 2021-06-10 18:42:27.921506: [TUN] [testClient] [Wintun] CreateAdapter: Creating adapter 2021-06-10 18:42:27.921506: [TUN] [testClient] [Wintun] CreateAdapter: Requested GUID is already in use: {928C5167-D802-9148-233E-28D4D15E1550}: Cannot create a file when that file already exists. (Code 0x000000B7) 2021-06-10 18:42:28.506458: [TUN] [testClient] Unable to create Wintun interface: Error creating interface: Cannot create a file when that file already exists. 2021-06-10 18:42:28.506524: [TUN] [testClient] Shutting down ... dead from here onward I acknowledge that killing the service PID is not the right way to stop wireguard, but it does seem to clean up pretty well (routes, etc) so I did not expect it to stay "broken". Please let me know if there's something I can do to get more info about any of the above errors. Thanks, -Eric From rcmcdonald91 at gmail.com Tue Jun 15 15:07:33 2021 From: rcmcdonald91 at gmail.com (Christian McDonald) Date: Tue, 15 Jun 2021 11:07:33 -0400 Subject: wg syncconf (and setconf) error when one or more endpoints is unresolvable In-Reply-To: References: Message-ID: Lonnie, Thanks for the quick response and the trivial fix! This is perfect Best, Christian On Tue, Jun 15, 2021 at 9:24 AM Lonnie Abelbeck wrote: > > > > On Jun 15, 2021, at 5:52 AM, Jason A. Donenfeld wrote: > > > > This is intended behavior. DNS resolution happens at config parsing time. > > Christian, > > While I appreciate Jason's strict DNS requirement, for the last 2.5 years our project has implemented a trivial patch [1] to ignore endpoint DNS failure. On a DNS failure, essentially ignoring the (optional) Endpoint= dns-hostname peer entry. > > This has worked well for our use case. WireGuard always starts. > > Lonnie > > [1] Ignore endpoint DNS failure > https://github.com/astlinux-project/astlinux/blob/master/package/wireguard-tools/wireguard-tools-0001-ignore-endpoint-dns-failure.patch > -- R. Christian McDonald E: rcmcdonald91 at gmail.com From Jason at zx2c4.com Tue Jun 15 15:22:42 2021 From: Jason at zx2c4.com (Jason A. Donenfeld) Date: Tue, 15 Jun 2021 17:22:42 +0200 Subject: wg syncconf (and setconf) error when one or more endpoints is unresolvable In-Reply-To: References: Message-ID: Hi Christian, I don't condone shipping patched binaries to your users, and I won't provide support for that here. What I'd recommend instead, if you want really fine grained control over DNS resolution, is to just resolve your DNS names prior to calling wg(8), and then apply whatever policy you want to the results of that prior resolution step, such as retries, discards, fallbacks, and so forth. Jason From daniel at makrotopia.org Wed Jun 16 13:24:57 2021 From: daniel at makrotopia.org (Daniel Golle) Date: Wed, 16 Jun 2021 15:24:57 +0200 Subject: passing-through TOS/DSCP marking Message-ID: Hi everyone, First of all, I'm sorry if this topic has been dealt with before and I've just not been able to find the answer in the archives. I'm currently helping to setup a bunch of Wireguard links for a non-profit humanitarian organization (using OpenWrt, ie. using the Linux implementation on both ends). As these tunnels are carrying a diverse mix of data, including VoIP, I was wondering if Wireguard passes-through the TOS/DSCP marking of tunneled packages to the outer package's headers, similar to what the 'passtos' option of OpenVPN is doing. I would assume this is true by default, but I haven't found any information about it, and as VoIP performance under load isn't as great as I was expecting, I'm now writing to you here, hoping for a meaningful answer which will preserve me from having to read the actual code... Thank you for all the great work! Best regards Daniel From harri at afaics.de Wed Jun 16 13:30:28 2021 From: harri at afaics.de (Harald Dunkel) Date: Wed, 16 Jun 2021 15:30:28 +0200 Subject: running wireguard company-wide? Message-ID: <87tulyos23.fsf@sylvester.afaics.de> Hi folks, how can I bind the private key to a specific host, eg. an office Macbook? Esp on MacOS the private key is in plain sight in the wireguard GUI, making it easy to reuse it for other, unintended purposes. For private installations this might be OK, but for a company-wide deployment this is a major headache. Every helpful hint is highly appreciated Harri From Jason at zx2c4.com Wed Jun 16 16:28:12 2021 From: Jason at zx2c4.com (Jason A. Donenfeld) Date: Wed, 16 Jun 2021 18:28:12 +0200 Subject: passing-through TOS/DSCP marking In-Reply-To: References: Message-ID: WireGuard does not copy the inner DSCP mark to the outside, aside from the ECN bits, in order to avoid a data leak. Jason From daniel at makrotopia.org Wed Jun 16 19:26:10 2021 From: daniel at makrotopia.org (Daniel Golle) Date: Wed, 16 Jun 2021 21:26:10 +0200 Subject: passing-through TOS/DSCP marking In-Reply-To: References: Message-ID: Hi Jason, On Wed, Jun 16, 2021 at 06:28:12PM +0200, Jason A. Donenfeld wrote: > WireGuard does not copy the inner DSCP mark to the outside, aside from > the ECN bits, in order to avoid a data leak. That's a very valid argument. However, from my experience now, Wireguard is not suitable for VoIP/RTP data (minimize-delay) being sent through the same tunnel as TCP bulk (maximize-throughput) traffic in bandwidth constraint and/or high-latency environments, as that ruins the VoIP calls to the degree of not being understandable. ECN helps quite a bit when it comes to avoid packet drops for TCP traffic, but that's not enough to avoid high jitter and drops for RTP/UDP traffic at the same time. I thought about ways to improve that and wonder what you would suggest. My ideas are: * have different tunnels depending on inner DSCP bits and mark them accordingly on the outside. => we already got multiple tunnels and that would double the number. * mark outer packets with DSCP bits based on their size. VoIP RTP/UDP packets are typically "medium sized" while TCP packets typically max out the MTU. => we would not leak information, but that assumption may not always be true * patch wireguard kernel code to allow preserving inner DSCP bits. => even only having 2 differentl classes of traffic (critical vs. bulk) would already help a lot... What do you think? Any other ideas? Cheers Daniel From toke at toke.dk Wed Jun 16 23:33:12 2021 From: toke at toke.dk (Toke =?utf-8?Q?H=C3=B8iland-J=C3=B8rgensen?=) Date: Thu, 17 Jun 2021 01:33:12 +0200 Subject: passing-through TOS/DSCP marking In-Reply-To: References: Message-ID: <87v96dpepz.fsf@toke.dk> Daniel Golle writes: > Hi Jason, > > On Wed, Jun 16, 2021 at 06:28:12PM +0200, Jason A. Donenfeld wrote: >> WireGuard does not copy the inner DSCP mark to the outside, aside from >> the ECN bits, in order to avoid a data leak. > > That's a very valid argument. > > However, from my experience now, Wireguard is not suitable for VoIP/RTP > data (minimize-delay) being sent through the same tunnel as TCP bulk > (maximize-throughput) traffic in bandwidth constraint and/or high-latency > environments, as that ruins the VoIP calls to the degree of not being > understandable. ECN helps quite a bit when it comes to avoid packet drops > for TCP traffic, but that's not enough to avoid high jitter and drops for > RTP/UDP traffic at the same time. > > I thought about ways to improve that and wonder what you would suggest. > My ideas are: > * have different tunnels depending on inner DSCP bits and mark them > accordingly on the outside. > => we already got multiple tunnels and that would double the number. > > * mark outer packets with DSCP bits based on their size. > VoIP RTP/UDP packets are typically "medium sized" while TCP packets > typically max out the MTU. > => we would not leak information, but that assumption may not always > be true > > * patch wireguard kernel code to allow preserving inner DSCP bits. > => even only having 2 differentl classes of traffic (critical vs. > bulk) would already help a lot... > > > What do you think? Any other ideas? Can you share a few more details about the network setup? I.e., where is the bottleneck link that requires this special treatment? If the bottleneck router is the same one that does the wireguard encapsulation (e.g., a home router with a slow uplink), you should be able to just use flow queueing (fq_codel or sch_cake) in place of your diffserv-based prioritisation and get most of the benefit: wireguard will save the packet hash before encapsulation, so any qdisc running on the same box can actually distinguish flows even on the encapsulated packets... -Toke From jaquilina at eagleeyet.net Thu Jun 17 03:59:16 2021 From: jaquilina at eagleeyet.net (Jonathan Aquilina) Date: Thu, 17 Jun 2021 03:59:16 +0000 Subject: running wireguard company-wide? In-Reply-To: <87tulyos23.fsf@sylvester.afaics.de> References: <87tulyos23.fsf@sylvester.afaics.de> Message-ID: Hi all, Just a long time follower of this list and very curious to try wireguard when I get the chance. On a Mac you have a keychain where you can store keys passwords certs, is there a reason why wireguard cannot take advantage of the keychain functionality? Regards, Jonathan -----Original Message----- From: WireGuard On Behalf Of Harald Dunkel Sent: 16 June 2021 15:30 To: wireguard at lists.zx2c4.com Subject: running wireguard company-wide? Hi folks, how can I bind the private key to a specific host, eg. an office Macbook? Esp on MacOS the private key is in plain sight in the wireguard GUI, making it easy to reuse it for other, unintended purposes. For private installations this might be OK, but for a company-wide deployment this is a major headache. Every helpful hint is highly appreciated Harri From daniel at makrotopia.org Thu Jun 17 09:41:02 2021 From: daniel at makrotopia.org (Daniel Golle) Date: Thu, 17 Jun 2021 11:41:02 +0200 Subject: passing-through TOS/DSCP marking In-Reply-To: <0102017a18f77a7e-85cc3154-dbac-4a9f-a0c5-acba247919a6-000000@eu-west-1.amazonses.com> References: <87v96dpepz.fsf@toke.dk> <0102017a18f77a7e-85cc3154-dbac-4a9f-a0c5-acba247919a6-000000@eu-west-1.amazonses.com> Message-ID: Hi Florent, On Thu, Jun 17, 2021 at 07:55:09AM +0000, Florent Daigniere wrote: > On Thu, 2021-06-17 at 01:33 +0200, Toke H?iland-J?rgensen wrote: > > Daniel Golle writes: > > > > > Hi Jason, > > > > > > On Wed, Jun 16, 2021 at 06:28:12PM +0200, Jason A. Donenfeld wrote: > > > > WireGuard does not copy the inner DSCP mark to the outside, aside > > > > from > > > > the ECN bits, in order to avoid a data leak. > > > > > > That's a very valid argument. > > > > > > However, from my experience now, Wireguard is not suitable for > > > VoIP/RTP > > > data (minimize-delay) being sent through the same tunnel as TCP bulk > > > (maximize-throughput) traffic in bandwidth constraint and/or high- > > > latency > > > environments, as that ruins the VoIP calls to the degree of not > > > being > > > understandable. ECN helps quite a bit when it comes to avoid packet > > > drops > > > for TCP traffic, but that's not enough to avoid high jitter and > > > drops for > > > RTP/UDP traffic at the same time. > > > > > > I thought about ways to improve that and wonder what you would > > > suggest. > > > My ideas are: > > > * have different tunnels depending on inner DSCP bits and mark them > > > accordingly on the outside. > > > => we already got multiple tunnels and that would double the > > > number. > > > > > > * mark outer packets with DSCP bits based on their size. > > > VoIP RTP/UDP packets are typically "medium sized" while TCP > > > packets > > > typically max out the MTU. > > > => we would not leak information, but that assumption may not > > > always > > > be true > > > > > > * patch wireguard kernel code to allow preserving inner DSCP bits. > > > => even only having 2 differentl classes of traffic (critical vs. > > > bulk) would already help a lot... > > > > > > > > > What do you think? Any other ideas? > > > > Can you share a few more details about the network setup? I.e., where > > is > > the bottleneck link that requires this special treatment? > > I can tell you about mine. WiFi in a congested environment: "voip on > mobile phones". WMM/802.11e uses the diffserv markings; most commercial > APs will do the right thing provided packets are marked appropriately. > > At the time I have sent patches (back in 2019) for both the golang and > linux implementation that turned it on by default. I believe that > Russell Strong further improved upon them by adding a knob (20190318 on > this mailing list). Thank you very much for the hint! This patch is exactly what I was looking for: https://lists.zx2c4.com/pipermail/wireguard/2019-March/004026.html Unfortunately it has not received a great amount of feedback back then. I'll try forward-porting and deploying it now, because to me it looks like the best solution money can buy :) > > Earlier this month I was approached by a NGO that was trying to do voip > over satlinks in between ships... there too, any solution has to involve > DSCP markings. This is quite exactly the scenario I'm working on :) Also here, we got multiple uplinks (2x VSAT, 1x 3G/4G/5G, 1x auxilary, e.g. WiFi client during shipyard times). All those uplinks are often congested and we can't do anything about their buffer bloat and as all of them rely on a shared medium, the total amount of available bandwidth varies quite a bit, which makes it hard to setup meaningful shaping. I did setup cake qdisc on the upstream links with generously estimated bandwidth settings, and that did not help a lot. However, all of those proprietary black-boxes do seem to care about the DSCP markings and that seems to be the way forward. Thank you! Daniel > > Florent > > From toke at toke.dk Thu Jun 17 12:24:29 2021 From: toke at toke.dk (Toke =?utf-8?Q?H=C3=B8iland-J=C3=B8rgensen?=) Date: Thu, 17 Jun 2021 14:24:29 +0200 Subject: passing-through TOS/DSCP marking In-Reply-To: References: <87v96dpepz.fsf@toke.dk> <0102017a18f77a7e-85cc3154-dbac-4a9f-a0c5-acba247919a6-000000@eu-west-1.amazonses.com> Message-ID: <87sg1gptky.fsf@toke.dk> Daniel Golle writes: > Hi Florent, > > On Thu, Jun 17, 2021 at 07:55:09AM +0000, Florent Daigniere wrote: >> On Thu, 2021-06-17 at 01:33 +0200, Toke H?iland-J?rgensen wrote: >> > Daniel Golle writes: >> > >> > > Hi Jason, >> > > >> > > On Wed, Jun 16, 2021 at 06:28:12PM +0200, Jason A. Donenfeld wrote: >> > > > WireGuard does not copy the inner DSCP mark to the outside, aside >> > > > from >> > > > the ECN bits, in order to avoid a data leak. >> > > >> > > That's a very valid argument. >> > > >> > > However, from my experience now, Wireguard is not suitable for >> > > VoIP/RTP >> > > data (minimize-delay) being sent through the same tunnel as TCP bulk >> > > (maximize-throughput) traffic in bandwidth constraint and/or high- >> > > latency >> > > environments, as that ruins the VoIP calls to the degree of not >> > > being >> > > understandable. ECN helps quite a bit when it comes to avoid packet >> > > drops >> > > for TCP traffic, but that's not enough to avoid high jitter and >> > > drops for >> > > RTP/UDP traffic at the same time. >> > > >> > > I thought about ways to improve that and wonder what you would >> > > suggest. >> > > My ideas are: >> > > * have different tunnels depending on inner DSCP bits and mark them >> > > accordingly on the outside. >> > > => we already got multiple tunnels and that would double the >> > > number. >> > > >> > > * mark outer packets with DSCP bits based on their size. >> > > VoIP RTP/UDP packets are typically "medium sized" while TCP >> > > packets >> > > typically max out the MTU. >> > > => we would not leak information, but that assumption may not >> > > always >> > > be true >> > > >> > > * patch wireguard kernel code to allow preserving inner DSCP bits. >> > > => even only having 2 differentl classes of traffic (critical vs. >> > > bulk) would already help a lot... >> > > >> > > >> > > What do you think? Any other ideas? >> > >> > Can you share a few more details about the network setup? I.e., where >> > is >> > the bottleneck link that requires this special treatment? >> >> I can tell you about mine. WiFi in a congested environment: "voip on >> mobile phones". WMM/802.11e uses the diffserv markings; most commercial >> APs will do the right thing provided packets are marked appropriately. >> >> At the time I have sent patches (back in 2019) for both the golang and >> linux implementation that turned it on by default. I believe that >> Russell Strong further improved upon them by adding a knob (20190318 on >> this mailing list). > > Thank you very much for the hint! > This patch is exactly what I was looking for: > https://lists.zx2c4.com/pipermail/wireguard/2019-March/004026.html > > Unfortunately it has not received a great amount of feedback back then. > I'll try forward-porting and deploying it now, because to me it looks > like the best solution money can buy :) I think you can achieve something similar using BPF filters, by relying on wireguard passing through the skb->hash value when encrypting. Simply attach a TC-BPF filter to the wireguard netdev, pull out the DSCP value and store it in a map keyed on skb->hash. Then, run a second BPF filter on the physical interface that shares that same map, lookup the DSCP value based on the skb->hash value, and rewrite the outer IP header. The read-side filter will need to use bpf_get_hash_recalc() to make sure the hash is calculated before the packet gets handed to wireguard, and it'll be subject to hash collisions, but I think it should generally work fairly well (for anything that's flow-based of course). And it can be done without patching wireguard itself :) -Toke From toke at toke.dk Thu Jun 17 20:54:55 2021 From: toke at toke.dk (Toke =?utf-8?Q?H=C3=B8iland-J=C3=B8rgensen?=) Date: Thu, 17 Jun 2021 22:54:55 +0200 Subject: passing-through TOS/DSCP marking In-Reply-To: References: <87v96dpepz.fsf@toke.dk> <0102017a18f77a7e-85cc3154-dbac-4a9f-a0c5-acba247919a6-000000@eu-west-1.amazonses.com> <87sg1gptky.fsf@toke.dk> Message-ID: <87lf78dxeo.fsf@toke.dk> Reid Rankin writes: > It can also be done in a shell script with nftables (maybe iptables too, > haven't tried) by taking advantage of fwmark passthrough. You can have one > rule that matches incoming outgoing packets (heh) with a certain dscp value > and marks them, and another rule that matches outgoing outgoing packets > with that mark and sets the DSCP bits back. The fwmark is not passed through wireguard, though, it's cleared during skb scrubbing: https://elixir.bootlin.com/linux/latest/source/net/core/skbuff.c#L5344 There's an fwmark config that you can set which will make wireguard apply a certain mark to all outgoing packets, but that has nothing to do with what was set on the inner packet... -Toke From toke at toke.dk Thu Jun 17 23:04:55 2021 From: toke at toke.dk (Toke =?utf-8?Q?H=C3=B8iland-J=C3=B8rgensen?=) Date: Fri, 18 Jun 2021 01:04:55 +0200 Subject: passing-through TOS/DSCP marking In-Reply-To: <87sg1gptky.fsf@toke.dk> References: <87v96dpepz.fsf@toke.dk> <0102017a18f77a7e-85cc3154-dbac-4a9f-a0c5-acba247919a6-000000@eu-west-1.amazonses.com> <87sg1gptky.fsf@toke.dk> Message-ID: <877disdre0.fsf@toke.dk> Toke H?iland-J?rgensen writes: > Daniel Golle writes: > >> Hi Florent, >> >> On Thu, Jun 17, 2021 at 07:55:09AM +0000, Florent Daigniere wrote: >>> On Thu, 2021-06-17 at 01:33 +0200, Toke H?iland-J?rgensen wrote: >>> > Daniel Golle writes: >>> > >>> > > Hi Jason, >>> > > >>> > > On Wed, Jun 16, 2021 at 06:28:12PM +0200, Jason A. Donenfeld wrote: >>> > > > WireGuard does not copy the inner DSCP mark to the outside, aside >>> > > > from >>> > > > the ECN bits, in order to avoid a data leak. >>> > > >>> > > That's a very valid argument. >>> > > >>> > > However, from my experience now, Wireguard is not suitable for >>> > > VoIP/RTP >>> > > data (minimize-delay) being sent through the same tunnel as TCP bulk >>> > > (maximize-throughput) traffic in bandwidth constraint and/or high- >>> > > latency >>> > > environments, as that ruins the VoIP calls to the degree of not >>> > > being >>> > > understandable. ECN helps quite a bit when it comes to avoid packet >>> > > drops >>> > > for TCP traffic, but that's not enough to avoid high jitter and >>> > > drops for >>> > > RTP/UDP traffic at the same time. >>> > > >>> > > I thought about ways to improve that and wonder what you would >>> > > suggest. >>> > > My ideas are: >>> > > * have different tunnels depending on inner DSCP bits and mark them >>> > > accordingly on the outside. >>> > > => we already got multiple tunnels and that would double the >>> > > number. >>> > > >>> > > * mark outer packets with DSCP bits based on their size. >>> > > VoIP RTP/UDP packets are typically "medium sized" while TCP >>> > > packets >>> > > typically max out the MTU. >>> > > => we would not leak information, but that assumption may not >>> > > always >>> > > be true >>> > > >>> > > * patch wireguard kernel code to allow preserving inner DSCP bits. >>> > > => even only having 2 differentl classes of traffic (critical vs. >>> > > bulk) would already help a lot... >>> > > >>> > > >>> > > What do you think? Any other ideas? >>> > >>> > Can you share a few more details about the network setup? I.e., where >>> > is >>> > the bottleneck link that requires this special treatment? >>> >>> I can tell you about mine. WiFi in a congested environment: "voip on >>> mobile phones". WMM/802.11e uses the diffserv markings; most commercial >>> APs will do the right thing provided packets are marked appropriately. >>> >>> At the time I have sent patches (back in 2019) for both the golang and >>> linux implementation that turned it on by default. I believe that >>> Russell Strong further improved upon them by adding a knob (20190318 on >>> this mailing list). >> >> Thank you very much for the hint! >> This patch is exactly what I was looking for: >> https://lists.zx2c4.com/pipermail/wireguard/2019-March/004026.html >> >> Unfortunately it has not received a great amount of feedback back then. >> I'll try forward-porting and deploying it now, because to me it looks >> like the best solution money can buy :) > > I think you can achieve something similar using BPF filters, by relying > on wireguard passing through the skb->hash value when encrypting. > > Simply attach a TC-BPF filter to the wireguard netdev, pull out the DSCP > value and store it in a map keyed on skb->hash. Then, run a second BPF > filter on the physical interface that shares that same map, lookup the > DSCP value based on the skb->hash value, and rewrite the outer IP > header. > > The read-side filter will need to use bpf_get_hash_recalc() to make sure > the hash is calculated before the packet gets handed to wireguard, and > it'll be subject to hash collisions, but I think it should generally > work fairly well (for anything that's flow-based of course). And it can > be done without patching wireguard itself :) Just for fun I implemented such a pair of eBPF filters, and tested that it does indeed work for preserving DSCP marks on a Wireguard tunnel. The PoC is here: https://github.com/xdp-project/bpf-examples/tree/master/preserve-dscp To try it out (you'll need a recent-ish kernel and clang version) run: git clone --recurse-submodules https://github.com/xdp-project/bpf-examples cd bpf-examples/preserve-dscp make ./preserve-dscp wg0 eth0 (assuming wg0 and eth0 are the wireguard and physical interfaces in question, respectively). To actually deploy this it would probably need a few tweaks; in particular the second filter that rewrites packets should probably check that the packets are actually part of the Wireguard tunnel in question (by parsing the UDP header and checking the source port) before writing anything to the packet. -Toke From Jason at zx2c4.com Fri Jun 18 12:24:29 2021 From: Jason at zx2c4.com (Jason A. Donenfeld) Date: Fri, 18 Jun 2021 14:24:29 +0200 Subject: passing-through TOS/DSCP marking In-Reply-To: <877disdre0.fsf@toke.dk> References: <87v96dpepz.fsf@toke.dk> <0102017a18f77a7e-85cc3154-dbac-4a9f-a0c5-acba247919a6-000000@eu-west-1.amazonses.com> <87sg1gptky.fsf@toke.dk> <877disdre0.fsf@toke.dk> Message-ID: Hey Toke, On Fri, Jun 18, 2021 at 1:05 AM Toke H?iland-J?rgensen wrote: > > I think you can achieve something similar using BPF filters, by relying > > on wireguard passing through the skb->hash value when encrypting. > > > > Simply attach a TC-BPF filter to the wireguard netdev, pull out the DSCP > > value and store it in a map keyed on skb->hash. Then, run a second BPF > > filter on the physical interface that shares that same map, lookup the > > DSCP value based on the skb->hash value, and rewrite the outer IP > > header. > > > > The read-side filter will need to use bpf_get_hash_recalc() to make sure > > the hash is calculated before the packet gets handed to wireguard, and > > it'll be subject to hash collisions, but I think it should generally > > work fairly well (for anything that's flow-based of course). And it can > > be done without patching wireguard itself :) > > Just for fun I implemented such a pair of eBPF filters, and tested that > it does indeed work for preserving DSCP marks on a Wireguard tunnel. The > PoC is here: > > https://github.com/xdp-project/bpf-examples/tree/master/preserve-dscp > > To try it out (you'll need a recent-ish kernel and clang version) run: > > git clone --recurse-submodules https://github.com/xdp-project/bpf-examples > cd bpf-examples/preserve-dscp > make > ./preserve-dscp wg0 eth0 > > (assuming wg0 and eth0 are the wireguard and physical interfaces in > question, respectively). > > To actually deploy this it would probably need a few tweaks; in > particular the second filter that rewrites packets should probably check > that the packets are actually part of the Wireguard tunnel in question > (by parsing the UDP header and checking the source port) before writing > anything to the packet. > > -Toke That is a super cool approach. Thanks for writing that! Sounds like a good approach, and one pretty easy to deploy, without the need to patch kernels and such. Also, nice usage of BPF_MAP_TYPE_LRU_HASH for this. Daniel -- can you let the list know if this works for your use case? Jason From nextgens at freenetproject.org Thu Jun 17 07:55:09 2021 From: nextgens at freenetproject.org (Florent Daigniere) Date: Thu, 17 Jun 2021 07:55:09 +0000 Subject: passing-through TOS/DSCP marking In-Reply-To: <87v96dpepz.fsf@toke.dk> References: <87v96dpepz.fsf@toke.dk> Message-ID: <0102017a18f77a7e-85cc3154-dbac-4a9f-a0c5-acba247919a6-000000@eu-west-1.amazonses.com> On Thu, 2021-06-17 at 01:33 +0200, Toke H?iland-J?rgensen wrote: > Daniel Golle writes: > > > Hi Jason, > > > > On Wed, Jun 16, 2021 at 06:28:12PM +0200, Jason A. Donenfeld wrote: > > > WireGuard does not copy the inner DSCP mark to the outside, aside > > > from > > > the ECN bits, in order to avoid a data leak. > > > > That's a very valid argument. > > > > However, from my experience now, Wireguard is not suitable for > > VoIP/RTP > > data (minimize-delay) being sent through the same tunnel as TCP bulk > > (maximize-throughput) traffic in bandwidth constraint and/or high- > > latency > > environments, as that ruins the VoIP calls to the degree of not > > being > > understandable. ECN helps quite a bit when it comes to avoid packet > > drops > > for TCP traffic, but that's not enough to avoid high jitter and > > drops for > > RTP/UDP traffic at the same time. > > > > I thought about ways to improve that and wonder what you would > > suggest. > > My ideas are: > > * have different tunnels depending on inner DSCP bits and mark them > > accordingly on the outside. > > => we already got multiple tunnels and that would double the > > number. > > > > * mark outer packets with DSCP bits based on their size. > > VoIP RTP/UDP packets are typically "medium sized" while TCP > > packets > > typically max out the MTU. > > => we would not leak information, but that assumption may not > > always > > be true > > > > * patch wireguard kernel code to allow preserving inner DSCP bits. > > => even only having 2 differentl classes of traffic (critical vs. > > bulk) would already help a lot... > > > > > > What do you think? Any other ideas? > > Can you share a few more details about the network setup? I.e., where > is > the bottleneck link that requires this special treatment? I can tell you about mine. WiFi in a congested environment: "voip on mobile phones". WMM/802.11e uses the diffserv markings; most commercial APs will do the right thing provided packets are marked appropriately. At the time I have sent patches (back in 2019) for both the golang and linux implementation that turned it on by default. I believe that Russell Strong further improved upon them by adding a knob (20190318 on this mailing list). Earlier this month I was approached by a NGO that was trying to do voip over satlinks in between ships... there too, any solution has to involve DSCP markings. Florent From and at mullvad.net Wed Jun 16 13:25:47 2021 From: and at mullvad.net (Andrej Mihajlov) Date: Wed, 16 Jun 2021 15:25:47 +0200 Subject: WireGuard is broken on iOS 15 beta In-Reply-To: References: <760B19F6-17E7-4276-B2E2-DFE07AB19323@mullvad.net> Message-ID: It was less about performance but more about security. Just wanted to share some good news! A little bit of background story: There was an article a little while ago that highlighted a traffic leak, when connections established before the VPN would continue going through the primary interface outside of the tunnel: https://protonvpn.com/blog/apple-ios-vulnerability-disclosure/ Then at some point around iOS 14.2, Apple introduced a kill-switch to address that issue (via VPN configuration flag). However it never really worked with WireGuard as the tunnel would always fail to establish the connection with peers. There is a thread on Apple forums with some of Apple engineers trying to figure out what?s going on: https://developer.apple.com/forums/thread/653116 Now it?s too early to say, but I just ran WireGuard with kill switch enabled on iOS 15 beta and the tunnel is magically up and running, which was still broken on iOS 14.3. I am going to test it on iOS 14.4 today too. So it looks like Apple did something to fix their SDK to make the kill switch work. Potential low effort improvement for WG, in practice it looks as such: let protocolConfig = NETunnelProviderProtocol() if #available(iOS 15.0, *) { protocolConfig.includeAllNetworks = true // kill-switch protocolConfig.excludeLocalNetworks = true // enable LAN networking } Cheers, Andrej > On 10 Jun 2021, at 13:51, Jason A. Donenfeld wrote: > > Patches with a proper performance comparison of the two techniques in > the description would be appreciated. > > Thanks, > Jason From daniel at makrotopia.org Mon Jun 21 12:36:03 2021 From: daniel at makrotopia.org (Daniel Golle) Date: Mon, 21 Jun 2021 14:36:03 +0200 Subject: passing-through TOS/DSCP marking In-Reply-To: References: <87v96dpepz.fsf@toke.dk> <0102017a18f77a7e-85cc3154-dbac-4a9f-a0c5-acba247919a6-000000@eu-west-1.amazonses.com> <87sg1gptky.fsf@toke.dk> <877disdre0.fsf@toke.dk> Message-ID: On Fri, Jun 18, 2021 at 02:24:29PM +0200, Jason A. Donenfeld wrote: > Hey Toke, > > On Fri, Jun 18, 2021 at 1:05 AM Toke H?iland-J?rgensen wrote: > > > I think you can achieve something similar using BPF filters, by relying > > > on wireguard passing through the skb->hash value when encrypting. > > > > > > Simply attach a TC-BPF filter to the wireguard netdev, pull out the DSCP > > > value and store it in a map keyed on skb->hash. Then, run a second BPF > > > filter on the physical interface that shares that same map, lookup the > > > DSCP value based on the skb->hash value, and rewrite the outer IP > > > header. > > > > > > The read-side filter will need to use bpf_get_hash_recalc() to make sure > > > the hash is calculated before the packet gets handed to wireguard, and > > > it'll be subject to hash collisions, but I think it should generally > > > work fairly well (for anything that's flow-based of course). And it can > > > be done without patching wireguard itself :) > > > > Just for fun I implemented such a pair of eBPF filters, and tested that > > it does indeed work for preserving DSCP marks on a Wireguard tunnel. The > > PoC is here: > > > > https://github.com/xdp-project/bpf-examples/tree/master/preserve-dscp > > > > To try it out (you'll need a recent-ish kernel and clang version) run: > > > > git clone --recurse-submodules https://github.com/xdp-project/bpf-examples > > cd bpf-examples/preserve-dscp > > make > > ./preserve-dscp wg0 eth0 > > > > (assuming wg0 and eth0 are the wireguard and physical interfaces in > > question, respectively). > > > > To actually deploy this it would probably need a few tweaks; in > > particular the second filter that rewrites packets should probably check > > that the packets are actually part of the Wireguard tunnel in question > > (by parsing the UDP header and checking the source port) before writing > > anything to the packet. > > > > -Toke > > That is a super cool approach. Thanks for writing that! Sounds like a > good approach, and one pretty easy to deploy, without the need to > patch kernels and such. > > Also, nice usage of BPF_MAP_TYPE_LRU_HASH for this. > > Daniel -- can you let the list know if this works for your use case? Turns out not exactly easy to deploy (on OpenWrt), as it depends on an extremely recent environment. I will try pushing to that direction, but it doesn't look like it's going to be ready very soon. In terms of toolchain: LLVM/Clang is a very bulky beast, I gave up on that and started working on integrating GCC-10's BPF target in our build system... In terms of kernel support: recent kernels don't build yet because of gelf_getsymshndx, so we got to update libelf first for that. Recent libelf doesn't seem to be an option yet on many of the build hosts we currently support (Darwin and such). In terms of library support: our build of libbpf comes from Linux release tarballs. There isn't yet a release supporting bpf_tc_attach, the easiest would be to wait for Linux 5.13 to be released. I (of course ;) also tried and spend almost a day looking for a quick-and-dirty path for temporary deployment, so I could at least give feedback -- bpf-examples also isn't exactly made to be cross-compiled manually, so I have failed with that as well so far. In general I still believe this is a great approach and I would really like to use it as soon as possible, even just to give you feedback. Cheers Daniel From toke at toke.dk Mon Jun 21 14:27:08 2021 From: toke at toke.dk (Toke =?utf-8?Q?H=C3=B8iland-J=C3=B8rgensen?=) Date: Mon, 21 Jun 2021 16:27:08 +0200 Subject: passing-through TOS/DSCP marking In-Reply-To: References: <87v96dpepz.fsf@toke.dk> <0102017a18f77a7e-85cc3154-dbac-4a9f-a0c5-acba247919a6-000000@eu-west-1.amazonses.com> <87sg1gptky.fsf@toke.dk> <877disdre0.fsf@toke.dk> Message-ID: <877dinths3.fsf@toke.dk> Daniel Golle writes: > On Fri, Jun 18, 2021 at 02:24:29PM +0200, Jason A. Donenfeld wrote: >> Hey Toke, >> >> On Fri, Jun 18, 2021 at 1:05 AM Toke H?iland-J?rgensen wrote: >> > > I think you can achieve something similar using BPF filters, by relying >> > > on wireguard passing through the skb->hash value when encrypting. >> > > >> > > Simply attach a TC-BPF filter to the wireguard netdev, pull out the DSCP >> > > value and store it in a map keyed on skb->hash. Then, run a second BPF >> > > filter on the physical interface that shares that same map, lookup the >> > > DSCP value based on the skb->hash value, and rewrite the outer IP >> > > header. >> > > >> > > The read-side filter will need to use bpf_get_hash_recalc() to make sure >> > > the hash is calculated before the packet gets handed to wireguard, and >> > > it'll be subject to hash collisions, but I think it should generally >> > > work fairly well (for anything that's flow-based of course). And it can >> > > be done without patching wireguard itself :) >> > >> > Just for fun I implemented such a pair of eBPF filters, and tested that >> > it does indeed work for preserving DSCP marks on a Wireguard tunnel. The >> > PoC is here: >> > >> > https://github.com/xdp-project/bpf-examples/tree/master/preserve-dscp >> > >> > To try it out (you'll need a recent-ish kernel and clang version) run: >> > >> > git clone --recurse-submodules https://github.com/xdp-project/bpf-examples >> > cd bpf-examples/preserve-dscp >> > make >> > ./preserve-dscp wg0 eth0 >> > >> > (assuming wg0 and eth0 are the wireguard and physical interfaces in >> > question, respectively). >> > >> > To actually deploy this it would probably need a few tweaks; in >> > particular the second filter that rewrites packets should probably check >> > that the packets are actually part of the Wireguard tunnel in question >> > (by parsing the UDP header and checking the source port) before writing >> > anything to the packet. >> > >> > -Toke >> >> That is a super cool approach. Thanks for writing that! Sounds like a >> good approach, and one pretty easy to deploy, without the need to >> patch kernels and such. >> >> Also, nice usage of BPF_MAP_TYPE_LRU_HASH for this. >> >> Daniel -- can you let the list know if this works for your use case? > > Turns out not exactly easy to deploy (on OpenWrt), as it depends on an > extremely recent environment. I will try pushing to that direction, but > it doesn't look like it's going to be ready very soon. > > In terms of toolchain: LLVM/Clang is a very bulky beast, I gave up on > that and started working on integrating GCC-10's BPF target in our build > system... I saw that, but I have no idea if GCC's BPF target support will support this. My tentative guess would be no, unfortunately :( An alternative to getting LLVM built as part of the OpenWrt toolchain is to just use the host clang to build the BPF binaries. It doesn't actually need to be cross-compiled with a special compiler, the BPF byte code format is the same on all architectures except for endianness, so just passing that to the host clang should theoretically be enough... > In terms of kernel support: recent kernels don't build yet because of > gelf_getsymshndx, so we got to update libelf first for that. Recent > libelf doesn't seem to be an option yet on many of the build hosts we > currently support (Darwin and such). > > In terms of library support: our build of libbpf comes from Linux > release tarballs. There isn't yet a release supporting bpf_tc_attach, > the easiest would be to wait for Linux 5.13 to be released. I used the libbpf TC loading support for convenience, but it's possible to load it using 'tc' as well without too much trouble (right now the userspace component sets a config variable before loading the program, but it can be restructured to not need that). Alternatively, the bpf-examples repository is setup with a libbpf submodule that it can link statically against, so you could use that for now? > I (of course ;) also tried and spend almost a day looking for a > quick-and-dirty path for temporary deployment, so I could at least give > feedback -- bpf-examples also isn't exactly made to be cross-compiled > manually, so I have failed with that as well so far. Heh, no, it isn't, really. Anything in particular you need to make this easier? We already added some bits to xdp-tools for supporting cross-compilation (and that shares some lineage with bpf-examples), so porting those over should not be too difficult. See: https://github.com/xdp-project/xdp-tools/pull/78 and https://github.com/xdp-project/xdp-tools/issues/74 Unfortunately I don't have a lot of time to poke more at this right now, but feel free to open up an issue / pull request to the bpf-examples repository with any changes you need :) -Toke From daniel at makrotopia.org Wed Jun 30 17:23:48 2021 From: daniel at makrotopia.org (Daniel Golle) Date: Wed, 30 Jun 2021 18:23:48 +0100 Subject: passing-through TOS/DSCP marking In-Reply-To: <877dinths3.fsf@toke.dk> References: <87v96dpepz.fsf@toke.dk> <0102017a18f77a7e-85cc3154-dbac-4a9f-a0c5-acba247919a6-000000@eu-west-1.amazonses.com> <87sg1gptky.fsf@toke.dk> <877disdre0.fsf@toke.dk> <877dinths3.fsf@toke.dk> Message-ID: Hi Toke, On Mon, Jun 21, 2021 at 04:27:08PM +0200, Toke H?iland-J?rgensen wrote: > Daniel Golle writes: > > > On Fri, Jun 18, 2021 at 02:24:29PM +0200, Jason A. Donenfeld wrote: > >> Hey Toke, > >> > >> On Fri, Jun 18, 2021 at 1:05 AM Toke H?iland-J?rgensen wrote: > >> > > I think you can achieve something similar using BPF filters, by relying > >> > > on wireguard passing through the skb->hash value when encrypting. > >> > > > >> > > Simply attach a TC-BPF filter to the wireguard netdev, pull out the DSCP > >> > > value and store it in a map keyed on skb->hash. Then, run a second BPF > >> > > filter on the physical interface that shares that same map, lookup the > >> > > DSCP value based on the skb->hash value, and rewrite the outer IP > >> > > header. > >> > > > >> > > The read-side filter will need to use bpf_get_hash_recalc() to make sure > >> > > the hash is calculated before the packet gets handed to wireguard, and > >> > > it'll be subject to hash collisions, but I think it should generally > >> > > work fairly well (for anything that's flow-based of course). And it can > >> > > be done without patching wireguard itself :) > >> > > >> > Just for fun I implemented such a pair of eBPF filters, and tested that > >> > it does indeed work for preserving DSCP marks on a Wireguard tunnel. The > >> > PoC is here: > >> > > >> > https://github.com/xdp-project/bpf-examples/tree/master/preserve-dscp > >> > > >> > To try it out (you'll need a recent-ish kernel and clang version) run: > >> > > >> > git clone --recurse-submodules https://github.com/xdp-project/bpf-examples > >> > cd bpf-examples/preserve-dscp > >> > make > >> > ./preserve-dscp wg0 eth0 > >> > > >> > (assuming wg0 and eth0 are the wireguard and physical interfaces in > >> > question, respectively). > >> > > >> > To actually deploy this it would probably need a few tweaks; in > >> > particular the second filter that rewrites packets should probably check > >> > that the packets are actually part of the Wireguard tunnel in question > >> > (by parsing the UDP header and checking the source port) before writing > >> > anything to the packet. > >> > > >> > -Toke > >> > >> That is a super cool approach. Thanks for writing that! Sounds like a > >> good approach, and one pretty easy to deploy, without the need to > >> patch kernels and such. > >> > >> Also, nice usage of BPF_MAP_TYPE_LRU_HASH for this. > >> > >> Daniel -- can you let the list know if this works for your use case? > > > > Turns out not exactly easy to deploy (on OpenWrt), as it depends on an > > extremely recent environment. I will try pushing to that direction, but > > it doesn't look like it's going to be ready very soon. > > > > In terms of toolchain: LLVM/Clang is a very bulky beast, I gave up on > > that and started working on integrating GCC-10's BPF target in our build > > system... > > I saw that, but I have no idea if GCC's BPF target support will support > this. My tentative guess would be no, unfortunately :( Probably you are right. When building the BPF object with GCC, the result is: root at OpenWrt:/usr/lib/bpf# preserve-dscp wg0 eth0 libbpf: elf: skipping unrecognized data section(4) .stab libbpf: elf: skipping relo section(5) .rel.stab for section(4) .stab libbpf: elf: skipping unrecognized data section(13) .comment libbpf: BTF is required, but is missing or corrupted. Couldn't open file: preserve_dscp_kern.o Using the LLVM/Clang compiled object also doesn't work: root at OpenWrt:/usr/lib/bpf# preserve-dscp wg0 eth0 libbpf: Error in bpf_create_map_xattr(flow_dscps):Operation not permitted(-1). Retrying without BTF. libbpf: map 'flow_dscps': failed to create: Operation not permitted(-1) libbpf: permission error while running as root; try raising 'ulimit -l'? current value: 512.0 KiB libbpf: failed to load object 'preserve_dscp_kern.o' Failed to load object Probably Kernel 5.4.124 is too old...? > > An alternative to getting LLVM built as part of the OpenWrt toolchain is > to just use the host clang to build the BPF binaries. It doesn't > actually need to be cross-compiled with a special compiler, the BPF byte > code format is the same on all architectures except for endianness, so > just passing that to the host clang should theoretically be enough... I believe that having a way to build BPF objects compatible with the target built-into our toolchain would be a huge step forward. And given that gcc already get's pretty far, I think it'd be worth fixing/patching what ever is missing (I haven't even tried GCC-11 yet) Find my staging tree including 'preserve-dscp' ready to play with: https://git.openwrt.org/?p=openwrt/staging/dangole.git;a=shortlog;h=refs/heads/gcc10-bpf Select 'Enable experimental features by default', but note that toolchain doesn't build when selecting Linux 5.10 for x86, so you need to un-select 'Use testing Kernel' if building for x86. And have a look at the patch for allow building bpf-examples BPF objects with GCC in package/network/utils/bpf-examples/patches > > > In terms of kernel support: recent kernels don't build yet because of > > gelf_getsymshndx, so we got to update libelf first for that. Recent > > libelf doesn't seem to be an option yet on many of the build hosts we > > currently support (Darwin and such). > > > > In terms of library support: our build of libbpf comes from Linux > > release tarballs. There isn't yet a release supporting bpf_tc_attach, > > the easiest would be to wait for Linux 5.13 to be released. > > I used the libbpf TC loading support for convenience, but it's possible > to load it using 'tc' as well without too much trouble (right now the > userspace component sets a config variable before loading the program, > but it can be restructured to not need that). > > Alternatively, the bpf-examples repository is setup with a libbpf > submodule that it can link statically against, so you could use that for > now? I've updated to 5.13 + patches on top, so now it builds :) Library-embedding is a no-go for OpenWrt. Having different ABI-versions of libraries installed simultanously works, so we can just ship with a more recent version of libbpf. > > > I (of course ;) also tried and spend almost a day looking for a > > quick-and-dirty path for temporary deployment, so I could at least give > > feedback -- bpf-examples also isn't exactly made to be cross-compiled > > manually, so I have failed with that as well so far. > > Heh, no, it isn't, really. Anything in particular you need to make this > easier? We already added some bits to xdp-tools for supporting > cross-compilation (and that shares some lineage with bpf-examples), so > porting those over should not be too difficult. I found my way around, see the packaging for bpf-examples in the tree (link above, at path stated above) > > See: https://github.com/xdp-project/xdp-tools/pull/78 and > https://github.com/xdp-project/xdp-tools/issues/74 > > Unfortunately I don't have a lot of time to poke more at this right now, > but feel free to open up an issue / pull request to the bpf-examples > repository with any changes you need :) I guess I'll just go ahead then and package xdp-tools :) Cheers Daniel > > -Toke From toke at toke.dk Wed Jun 30 20:55:09 2021 From: toke at toke.dk (Toke =?utf-8?Q?H=C3=B8iland-J=C3=B8rgensen?=) Date: Wed, 30 Jun 2021 22:55:09 +0200 Subject: passing-through TOS/DSCP marking In-Reply-To: References: <87v96dpepz.fsf@toke.dk> <0102017a18f77a7e-85cc3154-dbac-4a9f-a0c5-acba247919a6-000000@eu-west-1.amazonses.com> <87sg1gptky.fsf@toke.dk> <877disdre0.fsf@toke.dk> <877dinths3.fsf@toke.dk> Message-ID: <87h7hf139u.fsf@toke.dk> Daniel Golle writes: > Hi Toke, > > On Mon, Jun 21, 2021 at 04:27:08PM +0200, Toke H?iland-J?rgensen wrote: >> Daniel Golle writes: >> >> > On Fri, Jun 18, 2021 at 02:24:29PM +0200, Jason A. Donenfeld wrote: >> >> Hey Toke, >> >> >> >> On Fri, Jun 18, 2021 at 1:05 AM Toke H?iland-J?rgensen wrote: >> >> > > I think you can achieve something similar using BPF filters, by relying >> >> > > on wireguard passing through the skb->hash value when encrypting. >> >> > > >> >> > > Simply attach a TC-BPF filter to the wireguard netdev, pull out the DSCP >> >> > > value and store it in a map keyed on skb->hash. Then, run a second BPF >> >> > > filter on the physical interface that shares that same map, lookup the >> >> > > DSCP value based on the skb->hash value, and rewrite the outer IP >> >> > > header. >> >> > > >> >> > > The read-side filter will need to use bpf_get_hash_recalc() to make sure >> >> > > the hash is calculated before the packet gets handed to wireguard, and >> >> > > it'll be subject to hash collisions, but I think it should generally >> >> > > work fairly well (for anything that's flow-based of course). And it can >> >> > > be done without patching wireguard itself :) >> >> > >> >> > Just for fun I implemented such a pair of eBPF filters, and tested that >> >> > it does indeed work for preserving DSCP marks on a Wireguard tunnel. The >> >> > PoC is here: >> >> > >> >> > https://github.com/xdp-project/bpf-examples/tree/master/preserve-dscp >> >> > >> >> > To try it out (you'll need a recent-ish kernel and clang version) run: >> >> > >> >> > git clone --recurse-submodules https://github.com/xdp-project/bpf-examples >> >> > cd bpf-examples/preserve-dscp >> >> > make >> >> > ./preserve-dscp wg0 eth0 >> >> > >> >> > (assuming wg0 and eth0 are the wireguard and physical interfaces in >> >> > question, respectively). >> >> > >> >> > To actually deploy this it would probably need a few tweaks; in >> >> > particular the second filter that rewrites packets should probably check >> >> > that the packets are actually part of the Wireguard tunnel in question >> >> > (by parsing the UDP header and checking the source port) before writing >> >> > anything to the packet. >> >> > >> >> > -Toke >> >> >> >> That is a super cool approach. Thanks for writing that! Sounds like a >> >> good approach, and one pretty easy to deploy, without the need to >> >> patch kernels and such. >> >> >> >> Also, nice usage of BPF_MAP_TYPE_LRU_HASH for this. >> >> >> >> Daniel -- can you let the list know if this works for your use case? >> > >> > Turns out not exactly easy to deploy (on OpenWrt), as it depends on an >> > extremely recent environment. I will try pushing to that direction, but >> > it doesn't look like it's going to be ready very soon. >> > >> > In terms of toolchain: LLVM/Clang is a very bulky beast, I gave up on >> > that and started working on integrating GCC-10's BPF target in our build >> > system... >> >> I saw that, but I have no idea if GCC's BPF target support will support >> this. My tentative guess would be no, unfortunately :( > > Probably you are right. When building the BPF object with GCC, the > result is: > root at OpenWrt:/usr/lib/bpf# preserve-dscp wg0 eth0 > libbpf: elf: skipping unrecognized data section(4) .stab > libbpf: elf: skipping relo section(5) .rel.stab for section(4) .stab > libbpf: elf: skipping unrecognized data section(13) .comment > libbpf: BTF is required, but is missing or corrupted. > Couldn't open file: preserve_dscp_kern.o Hmm, for this example it should be possible to make it run without BTF. I'm only using that for the map definition, so that could be changed to the old format; you could try this patch: diff --git a/preserve-dscp/preserve_dscp_kern.c b/preserve-dscp/preserve_dscp_kern.c index 24120cb8a3ff..08248e1f0e41 100644 --- a/preserve-dscp/preserve_dscp_kern.c +++ b/preserve-dscp/preserve_dscp_kern.c @@ -9,12 +9,12 @@ * otherwise clean up stale entries. Instead, we just rely on the LRU mechanism * to evict old entries as the map fills up. */ -struct { - __uint(type, BPF_MAP_TYPE_LRU_HASH); - __type(key, __u32); - __type(value, __u8); - __uint(max_entries, 16384); -} flow_dscps SEC(".maps"); +struct bpf_map_def SEC("maps") flow_dscps = { + .type = BPF_MAP_TYPE_LRU_HASH, + .key_size = sizeof(__u32), + .value_size = sizeof(__u8), + .max_entries = 16384, +}; const volatile static int ip_only = 0; > Using the LLVM/Clang compiled object also doesn't work: > root at OpenWrt:/usr/lib/bpf# preserve-dscp wg0 eth0 > libbpf: Error in bpf_create_map_xattr(flow_dscps):Operation not permitted(-1). Retrying without BTF. > libbpf: map 'flow_dscps': failed to create: Operation not permitted(-1) > libbpf: permission error while running as root; try raising 'ulimit -l'? current value: 512.0 KiB > libbpf: failed to load object 'preserve_dscp_kern.o' > Failed to load object > > Probably Kernel 5.4.124 is too old...? Here I think the hint is in the error message ;) >> An alternative to getting LLVM built as part of the OpenWrt toolchain is >> to just use the host clang to build the BPF binaries. It doesn't >> actually need to be cross-compiled with a special compiler, the BPF byte >> code format is the same on all architectures except for endianness, so >> just passing that to the host clang should theoretically be enough... > > I believe that having a way to build BPF objects compatible with the > target built-into our toolchain would be a huge step forward. > And given that gcc already get's pretty far, I think it'd be worth > fixing/patching what ever is missing (I haven't even tried GCC-11 yet) For this example that might work (as noted above), but for other things BTF is a hard requirement, and I don't believe GCC supports that at all, sadly :( > Find my staging tree including 'preserve-dscp' ready to play with: > > https://git.openwrt.org/?p=openwrt/staging/dangole.git;a=shortlog;h=refs/heads/gcc10-bpf > > Select 'Enable experimental features by default', but note that toolchain > doesn't build when selecting Linux 5.10 for x86, so you need to un-select > 'Use testing Kernel' if building for x86. > And have a look at the patch for allow building bpf-examples BPF objects > with GCC in package/network/utils/bpf-examples/patches > > >> >> > In terms of kernel support: recent kernels don't build yet because of >> > gelf_getsymshndx, so we got to update libelf first for that. Recent >> > libelf doesn't seem to be an option yet on many of the build hosts we >> > currently support (Darwin and such). >> > >> > In terms of library support: our build of libbpf comes from Linux >> > release tarballs. There isn't yet a release supporting bpf_tc_attach, >> > the easiest would be to wait for Linux 5.13 to be released. >> >> I used the libbpf TC loading support for convenience, but it's possible >> to load it using 'tc' as well without too much trouble (right now the >> userspace component sets a config variable before loading the program, >> but it can be restructured to not need that). >> >> Alternatively, the bpf-examples repository is setup with a libbpf >> submodule that it can link statically against, so you could use that for >> now? > > I've updated to 5.13 + patches on top, so now it builds :) Alright, that works. > Library-embedding is a no-go for OpenWrt. Having different ABI-versions > of libraries installed simultanously works, so we can just ship with > a more recent version of libbpf. Yeah, I wasn't suggesting it as a permanent solution, just so you could test it out :) >> > I (of course ;) also tried and spend almost a day looking for a >> > quick-and-dirty path for temporary deployment, so I could at least give >> > feedback -- bpf-examples also isn't exactly made to be cross-compiled >> > manually, so I have failed with that as well so far. >> >> Heh, no, it isn't, really. Anything in particular you need to make this >> easier? We already added some bits to xdp-tools for supporting >> cross-compilation (and that shares some lineage with bpf-examples), so >> porting those over should not be too difficult. > > I found my way around, see the packaging for bpf-examples in the tree > (link above, at path stated above) Right, I see. >> >> See: https://github.com/xdp-project/xdp-tools/pull/78 and >> https://github.com/xdp-project/xdp-tools/issues/74 >> >> Unfortunately I don't have a lot of time to poke more at this right now, >> but feel free to open up an issue / pull request to the bpf-examples >> repository with any changes you need :) > > I guess I'll just go ahead then and package xdp-tools :) That would be awesome! xdp-tools will definitely need BTF, though, so I'm afraid it'll need to be compiled with LLVM at this stage... -Toke From audvare at gmail.com Sat Jun 19 21:10:57 2021 From: audvare at gmail.com (Andrew Udvare) Date: Sat, 19 Jun 2021 21:10:57 -0000 Subject: macOS app: DNS resolution failure Message-ID: <62264DB4-5919-4E1F-A162-80147A8FEECC@gmail.com> Hi, I am unable to connect to my tunnel that uses a name rather than IP. Every time I click activate, I get an alert "One or more endpoint domains could not be resolved." I can connect using a MacPorts-built copy of wg-quick in CLI. I can also connect if I put the mapping in /etc/hosts. I know I definitely have DNS working as nothing else has this issue. Is this a sandbox issue? I noticed the code uses getaddrinfo() and when I look in the logs after clicking activate, there are many errors like this: UNIX error exception: 17 The correct entitlements are present in the network extension entitlements. I cannot build my own copy at this time because I do not have a paid Apple Developer account. Thanks -- Andrew Udvare From bandry at ut1.org Mon Jun 28 13:44:58 2021 From: bandry at ut1.org (Bruno UT1) Date: Mon, 28 Jun 2021 13:44:58 -0000 Subject: Windows Client - Issue with Tray Icon In-Reply-To: <437b77e7-7e63-74c4-4447-713eccd42b59@ut-capitole.fr> References: <437b77e7-7e63-74c4-4447-713eccd42b59@ut-capitole.fr> Message-ID: <7587bd77-a81c-0488-90b5-eac7ed98a3db@ut1.org> Hi Wireguard Community! I think I found a small bug with Wireguard Windows Client. I made tests with version 0.3.14, 0.3.15 and 0.3.16 with the same behavior. My users are not administrators, so the registry key LimitedOperatorUI is set to 1. I made a script with wireguard.exe /installtunnelservice to automatically open the tunnel on some conditions. The script runs as administrator. When the script starts, the tunnel is opened and the status is Active in the main Window but the stays off on the tray. If the computer reboot, everything is back to normal but I can't force a reboot. Without the reboot, if the user try to activate the tunnel from the tray in fact the already opened tunnel is closed. After that, the tray has the good status again, but it disturbs the users. It's probably the fact that two distinct users are involved. Is there a possibility to synchronize the status? Thank you for this great VPN, Regards. Bruno From rowan at rowanthorpe.com Wed Jun 16 15:52:53 2021 From: rowan at rowanthorpe.com (Rowan Thorpe) Date: Wed, 16 Jun 2021 15:52:53 -0000 Subject: [PATCH] wg-quick: add embedded-friendly POSIX-shell version Message-ID: I had to do extreme configuring to get the latest libreCMC release onto my old (not officially supported any more, 4MB disk) router in order to have Wireguard available & usable, and although I succeeded it resulted in a very bare-bones kernel+OS, even by embedded standards. Aside from Bash obviously being missing (Busybox provides Ash) I also had to disable some of the usual Busybox & POSIX configuration options. I still wanted to use wg-quick though (and assumed others on very small systems may want it too) so I derived an "embedded-friendly posix shellscript" variant (with shims/monkeypatching for missing or deficient executables and for missing Bash-specific builtin functionality). Usually I wouldn't bother but this is one rare case where I think it is justified. I kept its structure as equivalent (and some code identical) to the linux Bash variant as possible, in the hopes it minimises added maintenance-burden. Because it has "lowest common denominator" requirements it can be parity-tested in the same environment as the other variants (but not fully tested there for its unique parts of course). I made it opt-in as one of the "wg-quick" variants by Make env-var (WITH_EMBEDDED + WITH_WGQUICK), but perhaps the preference is just to keep it separate, named something like "wg-quick-mini" in the contrib directory..? If so let me know & I'll re-send in that form. Signed-off-by: Rowan Thorpe --- src/Makefile | 12 +- src/wg-quick/posix.sh | 880 ++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 890 insertions(+), 2 deletions(-) create mode 100755 src/wg-quick/posix.sh diff --git a/src/Makefile b/src/Makefile index 7b8969a..50553e6 100644 --- a/src/Makefile +++ b/src/Makefile @@ -15,6 +15,7 @@ RUNSTATEDIR ?= /var/run WITH_BASHCOMPLETION ?= WITH_WGQUICK ?= WITH_SYSTEMDUNITS ?= +WITH_EMBEDDED ?= ifeq ($(WITH_BASHCOMPLETION),) ifneq ($(strip $(wildcard $(BASHCOMPDIR))),) @@ -28,6 +29,9 @@ endif ifneq ($(strip $(wildcard $(DESTDIR)/bin/bash)),) WITH_WGQUICK := yes endif +ifeq ($(WITH_EMBEDDED),yes) +WITH_WGQUICK := yes +endif endif ifeq ($(WITH_SYSTEMDUNITS),) ifneq ($(strip $(wildcard $(SYSTEMDUNITDIR))),) @@ -91,10 +95,14 @@ install: wg @[ "$(WITH_BASHCOMPLETION)" = "yes" ] || exit 0; \ install -v -d "$(DESTDIR)$(BASHCOMPDIR)" && install -v -m 0644 completion/wg.bash-completion "$(DESTDIR)$(BASHCOMPDIR)/wg" @[ "$(WITH_WGQUICK)" = "yes" ] || exit 0; \ - install -v -m 0755 wg-quick/$(PLATFORM).bash "$(DESTDIR)$(BINDIR)/wg-quick" && install -v -m 0700 -d "$(DESTDIR)$(SYSCONFDIR)/wireguard" + if [ "$(WITH_EMBEDDED)" = "yes" ]; then \ + install -v -m 0755 wg-quick/posix.sh "$(DESTDIR)$(BINDIR)/wg-quick"; \ + else \ + install -v -m 0755 wg-quick/$(PLATFORM).bash "$(DESTDIR)$(BINDIR)/wg-quick"; \ + fi && install -v -m 0700 -d "$(DESTDIR)$(SYSCONFDIR)/wireguard" @[ "$(WITH_WGQUICK)" = "yes" ] || exit 0; \ install -v -m 0644 man/wg-quick.8 "$(DESTDIR)$(MANDIR)/man8/wg-quick.8" - @[ "$(WITH_WGQUICK)" = "yes" -a "$(WITH_BASHCOMPLETION)" = "yes" ] || exit 0; \ + @[ "$(WITH_WGQUICK)" = "yes" -a "$(WITH_BASHCOMPLETION)" = "yes" -a "$(WITH_EMBEDDED)" != "yes" ] || exit 0; \ install -v -m 0644 completion/wg-quick.bash-completion "$(DESTDIR)$(BASHCOMPDIR)/wg-quick" @[ "$(WITH_WGQUICK)" = "yes" -a "$(WITH_SYSTEMDUNITS)" = "yes" ] || exit 0; \ install -v -d "$(DESTDIR)$(SYSTEMDUNITDIR)" && install -v -m 0644 systemd/* "$(DESTDIR)$(SYSTEMDUNITDIR)/" diff --git a/src/wg-quick/posix.sh b/src/wg-quick/posix.sh new file mode 100755 index 0000000..2ecdcbf --- /dev/null +++ b/src/wg-quick/posix.sh @@ -0,0 +1,880 @@ +#!/bin/sh +# -*- mode: sh; sh-indentation: 2; sh-basic-offset: 2; indent-tabs-mode: nil; fill-column: 100; coding: utf-8-unix; -*- +# +# SPDX-License-Identifier: GPL-2.0 +# +# Copyright (C) 2015-2020 Jason A. Donenfeld . All Rights Reserved. +# Austere posix/embedded variant derived by Rowan Thorpe , 2021. +# +# Sanity checked with: +# + shellcheck --check-sourced --external-sources --enable=all --shell=sh posix.sh +# + shfmt -d -ln posix -i 2 -ci posix.sh +# +# TODO: +# * Create local-vars drop-in functionality (without exploding complexity) to +# ensure recursion doesn't shadow vars. + +set -e + +## setup needed before function-definitions + +trap - EXIT +trap 'exit 1' HUP INT QUIT TERM + +# primitive exit-trap stack to keep things manageable +exit_trap() { + case "${1}" in + push) EXIT_TRAP="${2}${EXIT_TRAP:+${NL}${EXIT_TRAP}}" ;; + pop) EXIT_TRAP="$(printf '%s\n' "${EXIT_TRAP}" | tail -n +2)" ;; + *) exit 1 ;; + esac + #shellcheck disable=SC2064 + trap "${EXIT_TRAP:--}" EXIT +} + +# embedded systems without char-classes in "tr" need monkeypatching +if ! [ "$(printf 'aBcD' | tr '[:upper:]' '[:lower:]')" = 'abcd' ]; then + REAL_TR="$(command -v tr 2>/dev/null)" + tr() { + args='' + while [ "${#}" -ne 0 ]; do + case "${1}" in + '[:upper:]') + args="${args:+${args} }$(entity_save '[A-Z]')" + ;; + '[:lower:]') + args="${args:+${args} }$(entity_save '[a-z]')" + ;; + *) + args="${args:+${args} }$(entity_save "${1}")" + ;; + esac + shift + done + eval "${REAL_TR} ${args}" + unset args + } +fi + +# POSIX shells _may_ not have "type -p" so we need this drop-in +#shellcheck disable=SC2039 +if [ -n "$(type -p cat 2>/dev/null || :)" ]; then + type_p() { + type -p "${@}" + } +else + type_p() { + ret=0 + for arg; do + found=0 + for path in $(printf %s "${PATH-}" | tr ':' ' '); do + if [ -x "${path}/${arg}" ]; then + found=1 + break + fi + done + if [ "${found}" -eq 1 ]; then + printf '%s/%s' "${path}" "${arg}" + else + ret=1 + fi + done + unset arg found path + if [ "${ret}" -eq 0 ]; then + unset ret + return 0 + else + unset ret + return 1 + fi + } +fi + +# embedded systems without "stat" need this drop-in +if command -v stat >/dev/null 2>&1; then + stat_octal() { + stat -c '%04a' "${@}" + } +else + stat_octal() { + #shellcheck disable=SC2012 disable=SC2034 + ls -l "${@}" | + sed -ne ' + s/^[-dsbclp]\([-r]\)\([-w]\)\([-xsStT]\)\([-r]\)\([-w]\)\([-xsStT]\)\([-r]\)\([-w]\)\([-xsStT]\) .*$/\1 \2 \3 \4 \5 \6 \7 \8 \9/g + t P + b + : P + p + ' | + while read -r ur uw ux gr gw gx or ow ox; do + out='' + spc_sum=0 + for ctg in u g o; do + sum=0 + for perm in r w x; do + var="${ctg}${perm}" + eval "val=\"\${${var}}\"" + #shellcheck disable=SC2154 + case "${val}" in + r) exp=2 ;; + w) exp=1 ;; + s | t | x) exp=0 ;; + - | S | T) exp=-1 ;; + *) exit 1 ;; + esac + case "${val}" in + - | w | r | x) + spc_exp=-1 + ;; + S | s) + case "${var}" in + u*) spc_exp=2 ;; + g*) spc_exp=1 ;; + *) exit 1 ;; + esac + ;; + T | t) + case "${var}" in + o*) spc_exp=0 ;; + *) exit 1 ;; + esac + ;; + *) + exit 1 + ;; + esac + [ "${exp}" -lt 0 ] || + sum=$((sum + $((1 << exp)))) + [ "${spc_exp}" -lt 0 ] || + spc_sum=$((spc_sum + $((1 << spc_exp)))) + done + out="${out}$(printf %o "${sum}")" + done + printf '%o%s\n' "${spc_sum}" "${out}" + done + unset ur uw ux gr gw gx or ow ox ctg spc_sum out perm sum var val exp spc_exp + } +fi + +## + +e_body_save() { sed -e "s/'/'\\\\''/g"; } + +e_head_save() { sed -e "1s/^/'/"; } + +e_tail_save() { sed -e "\$s/\$/'/"; } + +e_save() { e_body_save | e_head_save | e_tail_save; } + +a_e_wrap() { sed -e '$s/$/ \\/'; } + +a_wrap() { sed -e '$s/$/\n /'; } + +entity_save() { printf '%s\n' "${1}" | e_save; } + +array_save() { + for i; do + entity_save "${i}" | a_e_wrap + done | + a_wrap + unset i +} + +array_append() { + orig_name="${1}" + shift + new=$(array_save "${@}") + eval " + eval \"set -- \${${orig_name}}\" + set -- \"\${@}\" ${new} + ${orig_name}=\$(array_save \"\${@}\") + " + unset orig_name new +} + +get_mtu() { + output="${1}" + existing_mtu="${2}" + shift 2 + mtu_match='' + dev_match='' + mtu_match="$(printf %s "${output}" | sed -ne 's:^.*\.*$:\1:; t P; b; : P; p; q')" + if [ -z "${mtu_match}" ]; then + dev_match="$(printf %s "${output}" | sed -ne 's:^.*\.*$:\1:; t P; b; : P; p; q')" + [ -z "${dev_match}" ] || + mtu_match="$(ip link show dev "${dev_match}" | sed -ne 's:^.*\.*$:\1:; t P; b; : P; p; q')" + fi + if [ -n "${mtu_match}" ] && + [ "${mtu_match}" -gt "${existing_mtu}" ]; then + printf %s "${mtu_match}" + else + printf %s "${existing_mtu}" + fi + unset output existing_mtu mtu_match dev_match +} + +## + +cmd() { + printf '[#] %s\n' "${*}" >&2 + "${@}" +} + +die() { + printf '%s: %s\n' "${PROGRAM}" "${*}" >&2 + exit 1 +} + +parse_options() { + interface_section=0 + line='' + key='' + value='' + stripped='' + v='' + header_line=0 + CONFIG_FILE="${1}" + #shellcheck disable=SC2003 + ! expr match "${CONFIG_FILE}" '[a-zA-Z0-9_=+.-]\{1,15\}$' >/dev/null || + CONFIG_FILE="${CONFIG_FILE_BASE}/${CONFIG_FILE}.conf" + [ -e "${CONFIG_FILE}" ] || + die "\`${CONFIG_FILE}' does not exist" + #shellcheck disable=SC2003 + expr match "${CONFIG_FILE}" '\(.*/\)\?\([a-zA-Z0-9_=+.-]\{1,15\}\)\.conf$' >/dev/null || + die 'The config file must be a valid interface name, followed by .conf' + CONFIG_FILE="$(readlink -f "${CONFIG_FILE}")" + if { + stat_octal "${CONFIG_FILE}" || : + stat_octal "$(printf %s "${CONFIG_FILE}" | sed -e 's:/[^/]*$::')" || : + } 2>/dev/null | grep -vq '0$'; then + printf 'Warning: `%s'\'' is world accessible\n' "${CONFIG_FILE}" >&2 + fi + INTERFACE="$(printf %s "${CONFIG_FILE}" | sed -e 's:^\(.*/\)\?\([^/.]\+\)\.conf$:\2:')" + while read -r line || [ -n "${line}" ]; do + stripped="$(printf %s "${line}" | sed -e 's:#.*$::; /^[[:blank:]]*$/d')" + key="$(printf %s "${stripped}" | sed -e 's#^[[:blank:]]*\([^=[:blank:]]\+\)[[:blank:]]*=.*$#\1#')" + case "${key}" in + '['*) + if [ "${key}" = '[Interface]' ]; then + interface_section=1 + else + interface_section=0 + fi + header_line=1 + ;; + *) + header_line=0 + ;; + esac + if [ "${header_line}" -eq 0 ] && [ "${interface_section}" -eq 1 ]; then + value="$( + printf %s "${stripped}" | + sed -e 's#^[^=]\+=[[:blank:]]*\([^[:blank:]]\(.*[^[:blank:]]\)\?\)\?[[:blank:]]*$#\1#' + )" + case "$(printf %s "${key}" | tr '[:upper:]' '[:lower:]')" in + address) + #shellcheck disable=SC2046 + array_append ADDRESSES $(printf %s "${value}" | tr ',' ' ') + continue + ;; + mtu) + MTU="${value}" + continue + ;; + dns) + for v in $(printf %s "${value}" | tr ',' ' '); do + #shellcheck disable=SC2003 + if expr match "${v}" '[0-9.]\+$' >/dev/null || expr match "${v}" '.*:.*$' >/dev/null; then + array_append DNS "${v}" + else + array_append DNS_SEARCH "${v}" + fi + done + continue + ;; + table) + TABLE="${value}" + continue + ;; + preup) + array_append PRE_UP "${value}" + continue + ;; + predown) + array_append PRE_DOWN "${value}" + continue + ;; + postup) + array_append POST_UP "${value}" + continue + ;; + postdown) + array_append POST_DOWN "${value}" + continue + ;; + saveconfig) + read_bool SAVE_CONFIG "${value}" + continue + ;; + *) + : + ;; + esac + fi + WG_CONFIG="${WG_CONFIG:+${WG_CONFIG}${NL}}${line}" + done <"${CONFIG_FILE}" + unset interface_section line key value stripped v header_line +} + +read_bool() { + case "${2}" in + true) eval "${1}=1" ;; + false) eval "${1}=0" ;; + *) die "\`${2}' is neither true nor false" ;; + esac +} + +#shellcheck disable=SC2120 +auto_su() { + if [ "${UID}" -ne 0 ]; then + eval "set -- ${ARGS}" + exec sudo -p "${PROGRAM} must be run as root. Please enter the password for %u to continue: " -- \ + "${SHELL:-/bin/sh}" -- "${SELF}" "${@}" + fi +} + +add_if() { + ret=0 + if ! cmd ip link add "${INTERFACE}" type wireguard; then + ret=${?} + ! [ -e /sys/module/wireguard ] && command -v "${WG_QUICK_USERSPACE_IMPLEMENTATION:-wireguard-go}" >/dev/null || + exit "${ret}" + printf '[!] Missing WireGuard kernel module. Falling back to slow userspace implementation.\n' >&2 + cmd "${WG_QUICK_USERSPACE_IMPLEMENTATION:-wireguard-go}" "${INTERFACE}" + fi + unset ret +} + +del_if() { + table='' + [ "${HAVE_SET_DNS-0}" -eq 0 ] || unset_dns + [ "${HAVE_SET_FIREWALL-0}" -eq 0 ] || remove_firewall + #shellcheck disable=SC2003 + if [ -z "${TABLE}" ] || + [ "x${TABLE}" = 'xauto' ] && + get_fwmark table && + expr match "$(wg show "${INTERFACE}" allowed-ips)" '.*/0\( .*\|'"${NL}"'.*\)\?$' >/dev/null; then + for proto in -4 -6; do + while :; do + case "$(ip "${proto}" rule show 2>/dev/null)" in + *"lookup ${table}"*) + cmd ip "${proto}" rule delete table "${table}" + ;; + *) + break + ;; + esac + done + while :; do + case "$(ip "${proto}" rule show 2>/dev/null)" in + *"from all lookup main suppress_prefixlength 0"*) + cmd ip "${proto}" rule delete table main suppress_prefixlength 0 + ;; + *) + break + ;; + esac + done + done + unset proto + fi + cmd ip link delete dev "${INTERFACE}" + unset table +} + +add_addr() { + case "${1}" in + *:*) proto=-6 ;; + *) proto=-4 ;; + esac + cmd ip "${proto}" address add "${1}" dev "${INTERFACE}" + unset proto +} + +set_mtu_up() { + mtu=0 + endpoint='' + v6_addr='' + if [ -n "${MTU}" ]; then + cmd ip link set mtu "${MTU}" up dev "${INTERFACE}" + else + wg show "${INTERFACE}" endpoints | { + while read -r _ endpoint; do + v6_addr="$( + printf %s "${endpoint}" | + sed -ne ' + s%^\[\([a-z0-9:.]\+\)\]:[0-9]\+$%\1% + t P + s%^\([a-z0-9:.]\+\):[0-9]\+$%\1% + t P + b + : P + p + ' + )" + [ -z "${v6_addr}" ] || + mtu="$(get_mtu "$(ip route get "${v6_addr}" || :)" "${mtu}")" + done + [ "${mtu}" -gt 0 ] || + mtu="$(get_mtu "$(ip route show default || :)" "${mtu}")" + [ "${mtu}" -gt 0 ] || mtu=1500 + cmd ip link set mtu $((mtu - 80)) up dev "${INTERFACE}" + } + fi + unset mtu endpoint v6_addr +} + +resolvconf_iface_prefix() { + if ! [ -f /etc/resolvconf/interface-order ]; then + iface='' + while read -r iface; do + #shellcheck disable=SC2003 + expr match "${iface}" '\([A-Za-z0-9-]\+\)\*$' >/dev/null || + continue + printf '%s\n' "${iface}" | + sed -e 's/\*\?$/./' + break + done /dev/null)" ] || + cmd ip "${proto}" route add "${1}" dev "${INTERFACE}" + ;; + esac + fi + unset proto +} + +get_fwmark() { + fwmark="$(wg show "${INTERFACE}" fwmark)" && + [ -n "${fwmark}" ] && + [ "x${fwmark}" != 'xoff' ] || + return 1 + eval "${1}=${fwmark}" + unset fwmark +} + +remove_firewall() { + if type_p nft >/dev/null; then + table='' + nftcmd='' + nft list tables 2>/dev/null | { + while read -r table; do + case "${table}" in + *" wg-quick-${INTERFACE}") + nftcmd="${nftcmd:+${nftcmd}${NL}}delete ${table}" + ;; + *) + : + ;; + esac + done + if [ -n "${nftcmd}" ]; then + printf '%s\n' "${nftcmd}" | + cmd nft -f + fi + } + unset table nftcmd + fi + if type_p iptables >/dev/null; then + iptables='' + for iptables in iptables ip6tables; do + "${iptables}-save" 2>/dev/null | { + restore='' + found=0 + line='' + while read -r line; do + case "${line}" in + \** | COMMIT | '-A '*'-m comment --comment "wg-quick(8) rule for '"${INTERFACE}"'"'*) + case "${line}" in + -A*) + found=1 + ;; + *) + : + ;; + esac + restore="${restore:+${restore}${NL}}-D${line#-A}" + ;; + *) + : + ;; + esac + done + [ "${found}" -ne 1 ] || + printf '%s\n' "${restore}" | + cmd "${iptables}-restore" -n + unset restore found line + } + done + unset iptables + fi +} + +add_default() { + table='' + line='' + proto='' + iptables='' + pf='' + marker='' + restore='' + nftable='' + nftcmd='' + if ! get_fwmark table; then + table=51820 + while [ -n "$(ip -4 route show table "${table}" 2>/dev/null)" ] || + [ -n "$(ip -6 route show table "${table}" 2>/dev/null)" ]; do + table=$((table + 1)) + done + cmd wg set "${INTERFACE}" fwmark "${table}" + fi + case "${1}" in + *:*) + proto='-6' + iptables='ip6tables' + pf='ip6' + ;; + *) + proto='-4' + iptables='iptables' + pf='ip' + ;; + esac + cmd ip "${proto}" route add "${1}" dev "${INTERFACE}" table "${table}" + cmd ip "${proto}" rule add not fwmark "${table}" table "${table}" + cmd ip "${proto}" rule add table main suppress_prefixlength 0 + + marker="-m comment --comment \"wg-quick(8) rule for ${INTERFACE}\"" + restore="*raw${NL}" + nftable="wg-quick-${INTERFACE}" + nftcmd="${nftcmd:+${nftcmd}${NL}}add table ${pf} ${nftable}" + nftcmd="${nftcmd:+${nftcmd}${NL}}add chain ${pf} ${nftable} preraw { type filter hook prerouting priority -300; }" + nftcmd="${nftcmd:+${nftcmd}${NL}}add chain ${pf} ${nftable} premangle { type filter hook prerouting priority -150; }" + nftcmd="${nftcmd:+${nftcmd}${NL}}add chain ${pf} ${nftable} postmangle { type filter hook postrouting priority -150; }" + ip -o "${proto}" addr show dev "${INTERFACE}" 2>/dev/null | { + match='' + while read -r line; do + match="$( + printf %s "${line}" | + sed -ne 's/^.*inet6\? \([0-9a-f:.]\+\)/[0-9]\+.*$/\1/; t P; b; : P; p' + )" + [ -n "${match}" ] || + continue + restore="${restore:+${restore}${NL}}-I PREROUTING ! -i ${INTERFACE} -d ${match} -m addrtype ! --src-type LOCAL -j DROP ${marker}" + nftcmd="${nftcmd:+${nftcmd}${NL}}add rule ${pf} ${nftable} preraw iifname != \"${INTERFACE}\" ${pf} daddr ${match} fib saddr type != local drop" + done + restore="${restore:+${restore}${NL}}COMMIT${NL}*mangle${NL}-I POSTROUTING -m mark --mark ${table} -p udp -j CONNMARK --save-mark ${marker}${NL}-I PREROUTING -p udp -j CONNMARK --restore-mark ${marker}${NL}COMMIT" + nftcmd="${nftcmd:+${nftcmd}${NL}}add rule ${pf} ${nftable} postmangle meta l4proto udp mark ${table} ct mark set mark" + nftcmd="${nftcmd:+${nftcmd}${NL}}add rule ${pf} ${nftable} premangle meta l4proto udp meta mark set ct mark" + ! [ "${proto}" = '-4' ] || + cmd sysctl -q net.ipv4.conf.all.src_valid_mark=1 + if type_p nft >/dev/null; then + printf '%s\n' "${nftcmd}" | + cmd nft -f + else + printf '%s\n' "${restore}" | + cmd "${iptables}-restore" -n + fi + unset match + } + HAVE_SET_FIREWALL=1 + unset table line proto iptables pf marker restore nftable nftcmd +} + +set_config() { + if [ -e /dev/stdin ]; then + printf '%s\n' "${WG_CONFIG}" | + cmd wg setconf "${INTERFACE}" /dev/stdin + else + tempfile="$(mktemp)" + exit_trap push "rm -f \"${tempfile}\"" + printf '%s\n' "${WG_CONFIG}" >"${tempfile}" + cmd wg setconf "${INTERFACE}" "${tempfile}" + rm -f "${tempfile}" + exit_trap pop + unset tempfile + fi +} + +save_config() { + old_umask='' + new_config='' + current_config='' + address='' + cmd='' + addr_match="$( + ip -all -brief address show dev "${INTERFACE}" | + sed -ne 's#^'"${INTERFACE}"' \+[A-Z]\+ \+\(.\+\)$#\1#; t P; b; : P; p' + )" + new_config='[Interface]' + for address in ${addr_match}; do + new_config="${new_config:+${new_config}${NL}}Address = ${address}" + done + { + resolvconf -l "$(resolvconf_iface_prefix)${INTERFACE}" 2>/dev/null || + cat "/etc/resolvconf/run/interface/$(resolvconf_iface_prefix)${INTERFACE}" 2>/dev/null + } | { + while read -r address; do + addr_match="$( + printf %s "${address}" | + sed -ne 's#^nameserver \([a-zA-Z0-9_=+:%.-]\+\)$#\1#; t P; b; : P; p' + )" + [ -z "${addr_match}" ] || + new_config="${new_config:+${new_config}${NL}}DNS = ${addr_match}" + done + if [ -n "${MTU}" ]; then + mtu_match="$( + ip link show dev "${INTERFACE}" | + sed -ne 's/^.*mtu \([0-9]\+\).*$/\1/; t P; b; : P; p' + )" + [ -z "${mtu_match}" ] || + new_config="${new_config:+${new_config}${NL}}MTU = ${mtu_match}" + fi + [ -z "${TABLE}" ] || + new_config="${new_config:+${new_config}${NL}}Table = ${TABLE}" + [ "${SAVE_CONFIG}" -eq 0 ] || + new_config="${new_config:+${new_config}${NL}}SaveConfig = true" + eval "set -- ${PRE_UP}" + for cmd; do + new_config="${new_config:+${new_config}${NL}}PreUp = ${cmd}" + done + eval "set -- ${POST_UP}" + for cmd; do + new_config="${new_config:+${new_config}${NL}}PostUp = ${cmd}" + done + eval "set -- ${PRE_DOWN}" + for cmd; do + new_config="${new_config:+${new_config}${NL}}PreDown = ${cmd}" + done + eval "set -- ${POST_DOWN}" + for cmd; do + new_config="${new_config:+${new_config}${NL}}PostDown = ${cmd}" + done + old_umask="$(umask)" + umask 077 + current_config="$(cmd wg showconf "${INTERFACE}")" + exit_trap push "rm -f \"${CONFIG_FILE}.tmp\"" + printf '%s\n' "${current_config}" | + sed -e "s#\\[Interface\\]\$#$( + printf %s "${new_config}" | + sed -e '$!s/$/\\n/' | + tr -d '\n' + )#" >"${CONFIG_FILE}.tmp" || + die 'Could not write configuration file' + sync "${CONFIG_FILE}.tmp" + mv "${CONFIG_FILE}.tmp" "${CONFIG_FILE}" || + die 'Could not move configuration file' + exit_trap pop + umask "${old_umask}" + unset new_config current_config old_umask cmd mtu_match addr_match address + } +} + +execute_hooks() { + for hook; do + hook="$( + printf %s "${hook}" | + sed -e "s^%i^${INTERFACE}^g" + )" + printf '[#] %s\n' "${hook}" >&2 + (eval "${hook}") + done + unset hook +} + +cmd_usage() { + cat >&2 <<-_EOF + Usage: ${PROGRAM} [ up | down | save | strip ] [ CONFIG_FILE | INTERFACE ] + + CONFIG_FILE is a configuration file, whose filename is the interface name + followed by \`.conf'. Otherwise, INTERFACE is an interface name, with + configuration found at ${CONFIG_FILE_BASE}/INTERFACE.conf. It is to be readable + by wg(8)'s \`setconf' sub-command, with the exception of the following additions + to the [Interface] section, which are handled by ${PROGRAM}: + + - Address: may be specified one or more times and contains one or more + IP addresses (with an optional CIDR mask) to be set for the interface. + - DNS: an optional DNS server to use while the device is up. + - MTU: an optional MTU for the interface; if unspecified, auto-calculated. + - Table: an optional routing table to which routes will be added; if + unspecified or \`auto', the default table is used. If \`off', no routes + are added. + - PreUp, PostUp, PreDown, PostDown: script snippets which will be executed + by bash(1) at the corresponding phases of the link, most commonly used + to configure DNS. The string \`%i' is expanded to INTERFACE. + - SaveConfig: if set to \`true', the configuration is saved from the current + state of the interface upon shutdown. + + See wg-quick(8) for more info and examples. + _EOF +} + +cmd_up() { + i='' + [ -z "$(ip link show dev "${INTERFACE}" 2>/dev/null)" ] || + die "\`${INTERFACE}' already exists" + exit_trap push 'del_if' + eval "execute_hooks ${PRE_UP}" + add_if + set_config + eval "set -- ${ADDRESSES}" + for i; do + add_addr "${i}" + done + set_mtu_up + set_dns + for i in $( + wg show "${INTERFACE}" allowed-ips | + while read -r _ j; do + for k in ${j}; do + #shellcheck disable=SC2003 + ! expr match "${k}" '[0-9a-z:.]\+/[0-9]\+$' >/dev/null || + printf '%s\n' "${k}" + done + done | + sort -nr -k 2 -t / + unset j k + ); do + add_route "${i}" + done + eval "execute_hooks ${POST_UP}" + unset i + exit_trap pop +} + +cmd_down() { + case " $(wg show interfaces) " in + *" ${INTERFACE} "*) : ;; + *) die "\`${INTERFACE}' is not a WireGuard interface" ;; + esac + eval "execute_hooks ${PRE_DOWN}" + [ "${SAVE_CONFIG}" -eq 0 ] || + save_config + del_if + unset_dns || : + remove_firewall || : + eval "execute_hooks ${POST_DOWN}" +} + +cmd_save() { + case " $(wg show interfaces) " in + *" ${INTERFACE} "*) : ;; + *) die "\`${INTERFACE}' is not a WireGuard interface" ;; + esac + save_config +} + +cmd_strip() { printf '%s\n' "${WG_CONFIG}"; } + +## + +EXIT_TRAP='' +LC_ALL=C +SELF="$(readlink -f "${0}")" +PATH="$(printf %s "${SELF}" | sed -e 's:/[^/]*$::'):${PATH}" +export LC_ALL PATH +[ -n "${UID-}" ] || UID="$(id -u)" +[ -n "${CONFIG_FILE_BASE}" ] || + CONFIG_FILE_BASE='/etc/wireguard' +NL=' +' +WG_CONFIG='' +INTERFACE='' +ADDRESSES=$(array_save) +MTU='' +DNS=$(array_save) +DNS_SEARCH=$(array_save) +TABLE='' +PRE_UP=$(array_save) +POST_UP=$(array_save) +PRE_DOWN=$(array_save) +POST_DOWN=$(array_save) +SAVE_CONFIG=0 +CONFIG_FILE='' +PROGRAM="$(printf %s "${0}" | sed -e 's:^.*/\([^/]*\)$:\1:')" +ARGS=$(array_save "${@}") +HAVE_SET_DNS=0 +HAVE_SET_FIREWALL=0 + +# ~~ function override insertion point ~~ + +case "${#}:${1}" in + 1:--help | 1:-h | 1:help) + cmd_usage + ;; + 2:up | 2:down | 2:save | 2:strip) + auto_su + parse_options "${2}" + case "${1}" in + up) + cmd_up + ;; + down) + cmd_down + ;; + save) + cmd_save + ;; + strip) + cmd_strip + ;; + *) + : + ;; + esac + ;; + *) + cmd_usage + exit 1 + ;; +esac -- 2.31.1 From tom.ty89 at gmail.com Fri Jun 25 05:06:24 2021 From: tom.ty89 at gmail.com (Tom Yan) Date: Fri, 25 Jun 2021 05:06:24 -0000 Subject: Wireguard MTU limitation on a server / forwarding peer Message-ID: Hi all, So I notice that wg-quick (and the Windows client) will use (the MTU of the default route interface - 80) as the MTU of the tunnel interface. Although I've read a mail about where the 80 comes from, I don't exactly know why the MTU of the tunnel interface needs to be that. I assume that it's for a reason like "to avoid encapsulated packets from being needed to be fragmented locally". I also notice that if on a server / forwarding peer, the MTU of the default route interface is smaller than the usual 1500, say 1460, and hence the MTU of the tunnel interface is capped at 1380, on its client peers I pretty much also need to cap the tunnel interface MTU at that (instead of letting it "falling back" to the usual 1420), seemingly have something to do with TCP MSS (which might be possible to workaround/fix with an ip/nftable rule instead I guess). My biggest doubt is however, whether I should "sync" the tunnel interface MTU of all peers. Say that on a / the server / forwarding peer is the usual 1420 by default, but it's known that it will serve / forward for client peers whose tunnel interface MTU would be as small as / needs to be capped at, say 1280. Should I set the tunnel interface MTU on the server / forwarding peer (and hence all of its other client peers) to 1280? (If it matters, say I don't need "client-to-client" communication.) (As you may have guessed, the "trigger" of this question / mail is that the default MTU used by the Android wireguard client is 1280. Although it's perhaps fine to bump it to 1420 on many devices, I do notice that at least on one of my phones the MTU of the cellular interface is apparently 1400.) Regards, Tom