<html>
<head>
<meta http-equiv="content-type" content="text/html; charset=utf-8">
</head>
<body text="#000000" bgcolor="#FFFFFF">
<p>I have a problem where I want to tunnel packets to a remote
subnet, but that remote subnet also includes the external IP of
the wireguard server. When that happens, the client goes into a
loop which soaks up 100% of CPU.<br>
</p>
<p>Here's the test setup:<br>
</p>
<p><br>
</p>
<p><tt> 10.12.255/24 | 10.12.254/24<br>
--+--------+----------------- Router ------+----------------</tt><tt><br>
</tt><tt> | .11 | .12</tt><tt> .1 .1 |
.115<br>
Target WG WG<br>
</tt><tt> Host server client</tt><br>
</p>
<p><br>
</p>
<p>* Wireguard "Server": Ubuntu 16.04, wireguard
0.0.20180620-wg1~xenial from PPA</p>
<p>* Wireguard "Client": macOS 10.12.6, wireguard-tools 0.0.20180625
installed from Homebrew</p>
<p>* Tunnel subnet is 10.12.0/24; the router has a static route to
10.12.0/24 via 10.12.255.12<br>
</p>
<p>* I want to reach hosts 10.12.255.x via the tunnel, such as
target host 10.12.255.11<br>
<br>
<tt>---- Server config ----</tt><tt><br>
</tt></p>
<p><tt>[Interface]</tt><tt><br>
</tt><tt>PrivateKey = <snip></tt><tt><br>
</tt><tt>ListenPort = XXXXX</tt><tt><br>
</tt><tt>Address = 10.12.0.1/24</tt><tt><br>
</tt><tt><br>
</tt><tt>[Peer]</tt><tt><br>
</tt><tt>PublicKey = GtMlTPv3tL++jG1eI2h7gJuuozgDp5F6iF+JUu0I/Fo=</tt><tt><br>
</tt><tt>AllowedIPs = 10.12.0.0/24</tt></p>
<p><tt>---- Client config ----</tt></p>
<p><tt>[Interface]</tt><tt><br>
</tt><tt>PrivateKey = <snip></tt><tt><br>
</tt><tt>ListenPort = YYYYY</tt><tt><br>
</tt><tt>Address = 10.12.0.2/24</tt><tt><br>
</tt><tt><br>
</tt><tt>[Peer]</tt><tt><br>
</tt><tt>PublicKey = 1kEwJOwzfMARwZG9H+A1QMfL3F76HZoxDhn1ciRdPnY=</tt><tt><br>
</tt><tt>EndPoint = 10.12.255.12:XXXXX</tt><tt><br>
</tt><tt>#AllowedIPs = 10.12.0.0/24 # (1)</tt><tt> ok<br>
</tt><tt>#AllowedIPs = 0.0.0.0/0 # (2) ok</tt><tt><br>
</tt><tt>#AllowedIPs = 10.12.0.0/16 # (3) fail</tt><tt><br>
</tt><tt>
AllowedIPs = 10.12.0.0/24, 10.12.255.0/24 # (4) fail</tt><tt><br>
</tt><tt>PersistentKeepalive = 22</tt></p>
<p>It works with (1) AllowedIPs = 10.12.0.0/24: the client can ping
the server 10.12.0.1. And it works with (2) AllowedIPs =
0.0.0.0/0: I can ping the target subnet (but also all my Internet
traffic is routed down the tunnel, which I don't want).<br>
</p>
<p>However with configuration (3) or (4), shortly after I do
"wg-quick up wg0", CPU load on the macOS client jumps to 100%, as
shown by Activity Monitor or top -u, and the fans spin up.</p>
<p><br>
</p>
<tt>bash-3.2# wg-quick up wg0</tt><tt><br>
</tt><tt>[#] wireguard-go utun</tt><tt><br>
</tt><tt>WARNING WARNING WARNING WARNING WARNING WARNING WARNING</tt><tt><br>
</tt><tt>W G</tt><tt><br>
</tt><tt>W This is alpha software. It will very likely not G</tt><tt><br>
</tt><tt>W do what it is supposed to do, and things may go G</tt><tt><br>
</tt><tt>W horribly wrong. You have been warned. Proceed G</tt><tt><br>
</tt><tt>W at your own risk. G</tt><tt><br>
</tt><tt>W G</tt><tt><br>
</tt><tt>WARNING WARNING WARNING WARNING WARNING WARNING WARNING</tt><tt><br>
</tt><tt>INFO: (utun1) 2018/07/01 10:56:26 Starting wireguard-go
version 0.0.20180613</tt><tt><br>
</tt><tt>[+] Interface for wg0 is utun1</tt><tt><br>
</tt><tt>[#] wg setconf utun1 /dev/fd/63</tt><tt><br>
</tt><tt>[#] ifconfig utun1 inet 10.12.0.2/24 10.12.0.2 alias</tt><tt><br>
</tt><tt>[#] ifconfig utun1 up</tt><tt><br>
</tt><tt>[#] route -q -n add -inet 10.12.255.0/24 -interface utun1</tt><tt><br>
</tt><tt>[#] route -q -n add -inet 10.12.0.0/24 -interface utun1</tt><tt><br>
</tt><tt>[+] Backgrounding route monitor</tt><tt><br>
</tt><br>
Within about 20 seconds, CPU usage jumps up to max. "wg" shows huge
amounts of traffic sent. The following are captured at
approximately 1 second intervals:<br>
<br>
<tt>bash-3.2# wg<br>
interface: utun1<br>
public key: GtMlTPv3tL++jG1eI2h7gJuuozgDp5F6iF+JUu0I/Fo=<br>
private key: (hidden)<br>
listening port: YYYYY<br>
<br>
peer: 1kEwJOwzfMARwZG9H+A1QMfL3F76HZoxDhn1ciRdPnY=<br>
endpoint: 10.12.255.12:XXXXX<br>
allowed ips: 10.12.0.0/24, 10.12.255.0/24<br>
latest handshake: 39 seconds ago<br>
transfer: 0 B received, <b>629.43 MiB sent</b><br>
persistent keepalive: every 22 seconds<br>
bash-3.2# wg<br>
interface: utun1<br>
public key: GtMlTPv3tL++jG1eI2h7gJuuozgDp5F6iF+JUu0I/Fo=<br>
private key: (hidden)<br>
listening port: YYYYY<br>
<br>
peer: 1kEwJOwzfMARwZG9H+A1QMfL3F76HZoxDhn1ciRdPnY=<br>
endpoint: 10.12.255.12:XXXXX<br>
allowed ips: 10.12.0.0/24, 10.12.255.0/24<br>
latest handshake: 41 seconds ago<br>
transfer: 0 B received, <b>694.98 MiB sent</b><br>
persistent keepalive: every 22 seconds<br>
bash-3.2# wg<br>
interface: utun1<br>
public key: GtMlTPv3tL++jG1eI2h7gJuuozgDp5F6iF+JUu0I/Fo=<br>
private key: (hidden)<br>
listening port: YYYYY<br>
<br>
peer: 1kEwJOwzfMARwZG9H+A1QMfL3F76HZoxDhn1ciRdPnY=<br>
endpoint: 10.12.255.12:XXXXX<br>
allowed ips: 10.12.0.0/24, 10.12.255.0/24<br>
latest handshake: 43 seconds ago<br>
transfer: 0 B received, <b>765.37 MiB sent</b><br>
persistent keepalive: every 22 seconds</tt><tt></tt><br>
<br>
But tcpdump shows only an initial exchange of packets:<br>
<br>
<tt>$ sudo tcpdump -i en0 -nn udp port YYYYY or udp port XXXXX</tt><tt><br>
</tt><tt>Password:</tt><tt><br>
</tt><tt>tcpdump: verbose output suppressed, use -v or -vv for full
protocol decode</tt><tt><br>
</tt><tt>listening on en0, link-type EN10MB (Ethernet), capture size
262144 bytes</tt><tt><br>
</tt><tt>11:06:07.340249 IP 10.12.254.115.YYYYY >
10.12.255.12.XXXXX: UDP, length 148</tt><tt><br>
</tt><tt>11:06:07.343687 IP 10.12.255.12.XXXXX >
10.12.254.115.YYYYY: UDP, length 92</tt><tt><br>
</tt><tt>11:06:07.344060 IP 10.12.254.115.YYYYY >
10.12.255.12.XXXXX: UDP, length 32</tt><tt><br>
</tt><tt><br>
</tt><br>
I suspect the problem is to do with the server external IP of
10.12.255.12 being within the AllowedIPs = 10.12.255.0/24 range.
While the tunnel is up, here is the routing table on the client:<br>
<br>
<tt>bash-3.2# netstat -rn</tt><tt><br>
</tt><tt>Routing tables</tt><tt><br>
</tt><tt><br>
</tt><tt>Internet:</tt><tt><br>
</tt><tt>Destination Gateway Flags
Refs Use Netif Expire</tt><tt><br>
</tt><tt>default 10.12.254.1 UGSc
15 0 en0</tt><tt><br>
</tt><tt>10.12/24 utun1 USc
0 0 utun1</tt><tt><br>
</tt><tt>10.12.0.2 10.12.0.2 UH
0 0 utun1</tt><tt><br>
</tt><tt>10.12.254/24 link#4 UCS
6 0 en0</tt><tt><br>
</tt><tt>10.12.254.1/32 link#4 UCS
1 0 en0</tt><tt><br>
</tt><tt>10.12.254.1 64:d1:54:xx:xx:xx UHLWIir
19 67 en0 1197</tt><tt><br>
</tt><tt>10.12.254.100 0:e:58:xx:xx:xx UHLWI
0 0 en0 1127</tt><tt><br>
</tt><tt>10.12.254.101 b8:e9:37:xx:x:xx UHLWI
0 0 en0 1125</tt><tt><br>
</tt><tt>10.12.254.103 0:e:58:xx:xx:x UHLWI
0 0 en0 1109</tt><tt><br>
</tt><tt>10.12.254.104 0:e:58:xx:xx:xx UHLWI
0 0 en0 1088</tt><tt><br>
</tt><tt>10.12.254.115/32 link#4 UCS
0 0 en0</tt><tt><br>
</tt><tt>10.12.254.117 44:d2:44:xx:xx:xx UHLWI
0 0 en0 1177</tt><tt><br>
</tt><tt>10.12.255/24 utun1 USc
1 0 utun1</tt><tt><br>
</tt><tt>127 127.0.0.1 UCS
0 0 lo0</tt><tt><br>
</tt><tt>127.0.0.1 127.0.0.1 UH 1
122329 lo0</tt><tt><br>
</tt><tt>169.254 link#4 UCS
0 0 en0</tt><tt><br>
</tt><tt>192.168.56 link#11 UCS
1 0 vboxnet</tt><tt><br>
</tt><tt>224.0.0/4 link#4 UmCS
2 0 en0</tt><tt><br>
</tt><tt>224.0.0.251 1:0:5e:0:0:fb UHmLWI
0 0 en0</tt><tt><br>
</tt><tt>239.255.255.250 1:0:5e:7f:ff:fa UHmLWI
0 12 en0</tt><tt><br>
</tt><tt>255.255.255.255/32 link#4 UCS
0 0 en0</tt><br>
<br>
That seems to be it: if it were to send encrypted packets to
10.12.255.12 during this time, then they would be sent back down
utun1 to be re-encrypted again.<br>
<br>
However, it does work fine with AllowedIPs = 0.0.0.0/0. Here is the
routing table when I do that:<br>
<br>
<tt>bash-3.2# netstat -rn</tt><tt><br>
</tt><tt>Routing tables</tt><tt><br>
</tt><tt><br>
</tt><tt>Internet:</tt><tt><br>
</tt><tt>Destination Gateway Flags
Refs Use Netif Expire</tt><tt><br>
</tt><tt>0/1 utun1 USc
4 1 utun1</tt><tt><br>
</tt><tt>default 10.12.254.1 UGSc
11 0 en0</tt><tt><br>
</tt><tt>10.12.0.2 10.12.0.2 UH
0 0 utun1</tt><tt><br>
</tt><tt>10.12.254/24 link#4 UCS
7 0 en0</tt><tt><br>
</tt><tt>10.12.254.1/32 link#4 UCS
1 0 en0</tt><tt><br>
</tt><tt>10.12.254.1 64:d1:54:xx:xx:xx UHLWIir
7 180 en0 1197</tt><tt><br>
</tt><tt>10.12.254.100 0:e:58:xx:xx:xx UHLWI
0 0 en0 963</tt><tt><br>
</tt><tt>10.12.254.101 b8:e9:37:xx:x:xx UHLWI
0 0 en0 961</tt><tt><br>
</tt><tt>10.12.254.103 0:e:58:xx:xx:x UHLWI
0 0 en0 945</tt><tt><br>
</tt><tt>10.12.254.104 0:e:58:xx:xx:xx UHLWI
0 0 en0 924</tt><tt><br>
</tt><tt>10.12.254.107 9c:4:eb:xx:xx:xx UHLWI
0 0 en0 708</tt><tt><br>
</tt><tt>10.12.254.115/32 link#4 UCS
0 0 en0</tt><tt><br>
</tt><tt>10.12.254.117 44:d2:44:xx:xx:xx UHLWI
0 0 en0 1163</tt><tt><br>
</tt><tt>10.12.254.255 ff:ff:ff:ff:ff:ff UHLWbI
0 3 en0</tt><tt><br>
</tt><b><tt>10.12.255.12 10.12.254.1 UGHS
2 93 en0</tt></b><b><tt><br>
</tt></b><tt>127 127.0.0.1 UCS
0 0 lo0</tt><tt><br>
</tt><tt>127.0.0.1 127.0.0.1 UH 1
122329 lo0</tt><tt><br>
</tt><tt>128.0/1 utun1 USc
1 0 utun1</tt><tt><br>
</tt><tt>169.254 link#4 UCS
0 0 en0</tt><tt><br>
</tt><tt>192.168.56 link#11 UCS
2 0 vboxnet</tt><tt><br>
</tt><tt>192.168.56.255 ff:ff:ff:ff:ff:ff UHLWbI
0 3 vboxnet</tt><tt><br>
</tt><tt>224.0.0/4 link#4 UmCS
2 0 en0</tt><tt><br>
</tt><tt>224.0.0.251 1:0:5e:0:0:fb UHmLWI
0 0 en0</tt><tt><br>
</tt><tt>239.255.255.250 1:0:5e:7f:ff:fa UHmLWI
0 76 en0</tt><tt><br>
</tt><tt>255.255.255.255/32 link#4 UCS
0 0 en0</tt><br>
<br>
I see that a specific host route has been added for 10.12.255.12.<br>
<br>
<tt>INFO: (utun1) 2018/07/01 11:30:01 Starting wireguard-go version
0.0.20180613</tt><tt><br>
</tt><tt>[+] Interface for wg0 is utun1</tt><tt><br>
</tt><tt>[#] wg setconf utun1 /dev/fd/63</tt><tt><br>
</tt><tt>[#] ifconfig utun1 inet 10.12.0.2/24 10.12.0.2 alias</tt><tt><br>
</tt><tt>[#] ifconfig utun1 up</tt><tt><br>
</tt><tt>[#] route -q -n add -inet 0.0.0.0/1 -interface utun1</tt><tt><br>
</tt><tt>[#] route -q -n add -inet 128.0.0.0/1 -interface utun1</tt><tt><br>
</tt><tt><b>[#] route -q -n add -inet 10.12.255.12 -gateway
10.12.254.1</b></tt><tt><b><br>
</b></tt><tt>[+] Backgrounding route monitor</tt><br>
<br>
And I find this is documented - the wg-quick manpage says:<br>
<br>
<i>"If one of those routes is the default route (0.0.0.0/0 or ::/0),
then it uses ip-rule(8) to handle overriding of the default
gateway."</i><br>
<br>
So I think the answer is straightforward: I would like this rule to
be added when the target IP is within any AllowedIPs subnet, not
just for 0.0.0.0/0. Would you agree?<br>
<br>
If I add this route manually, everything seems to work fine.<br>
<br>
Thanks,<br>
<br>
Brian Candler.<br>
<br>
</body>
</html>