What would cause Ubuntu 18.04 to perform different fragmentation when forwarding to WireGuard tunnel, depending on packet source?

[Sander Saares]

Hi!

I am troubleshooting some odd behavior and noticed that when my Ubuntu 18.04 server is routing packets between the internet and a WireGuard tunnel, there exist some odd differences between how different connections are handled that I cannot account for.

In both cases, I have a client (192.168.147.14) from the other side of the tunnel making an HTTPS connection. The IPv4 packets received from the internet have the "don't fragment" flag set, as is common. What I find is:


  *   When processing large packets from server A (159.148.147.196), Ubuntu responds with "needs fragmentation" and correctly gives the WireGuard MTU of 1420 as the next hop MTU.
  *   When processing large packets from server B (40.68.232.16), Ubuntu just does the fragmentation itself and forwards the packets no questions asked. Despite "don't fragment" bit being set.

I am unable to explain the difference in behavior. What am I missing here? Why might the two cases be handled differently by the operating system? I do not even know if this is WireGuard related - maybe it also occurs with non-WireGuard adapters.

My forwarding-relevant WireGuard configuration is simply the following:

PostUp = iptables -A FORWARD -i wg0 -j ACCEPT; iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE
PostDown = iptables -D FORWARD -i wg0 -j ACCEPT; iptables -t nat -D POSTROUTING -o eth0 -j MASQUERADE

I have not adjusted any relevant operating system settings from defaults other than allowing IP forwarding.

The initial packets of the connection that do not exceed the WireGuard MTU are correctly routed through the WireGuard tunnel in both cases.

Response from server A
Internet Protocol Version 4, Src: 159.148.147.196, Dst: 172.16.21.250
    0100 .... = Version: 4
    .... 0101 = Header Length: 20 bytes (5)
    Differentiated Services Field: 0x00 (DSCP: CS0, ECN: Not-ECT)
    Total Length: 3042
    Identification: 0x2839 (10297)
    Flags: 0x4000, Don't fragment
    ...0 0000 0000 0000 = Fragment offset: 0
    Time to live: 46
    Protocol: TCP (6)
    Header checksum: 0x237a [validation disabled]
    [Header checksum status: Unverified]
    Source: 159.148.147.196
    Destination: 172.16.21.250

In response, "fragmentation needed" is sent
Internet Protocol Version 4, Src: 172.16.21.250, Dst: 159.148.147.196
Internet Control Message Protocol
    Type: 3 (Destination unreachable)
    Code: 4 (Fragmentation needed)
    Checksum: 0x99ba [correct]
    [Checksum Status: Good]
    Unused: a8
    Length: 246
    [Length of original datagram: 984]
    MTU of next hop: 1420
    Internet Protocol Version 4, Src: 159.148.147.196, Dst: 172.16.21.250
    Transmission Control Protocol, Src Port: 443, Dst Port: 43254, Seq: 2041044714, Ack: 1154907605
    Transport Layer Security

Response from server B
Internet Protocol Version 4, Src: 40.68.232.16, Dst: 172.16.21.250
    0100 .... = Version: 4
    .... 0101 = Header Length: 20 bytes (5)
    Differentiated Services Field: 0x00 (DSCP: CS0, ECN: Not-ECT)
    Total Length: 5716
    Identification: 0x22e1 (8929)
    Flags: 0x4000, Don't fragment
    ...0 0000 0000 0000 = Fragment offset: 0
    Time to live: 119
    Protocol: TCP (6)
    Header checksum: 0xf863 [validation disabled]
    [Header checksum status: Unverified]
    Source: 40.68.232.16
    Destination: 172.16.21.250

In response, Ubuntu just fragments the packet anyway and pushes through the WG tunnel
22           0.081525663       172.16.21.250     192.168.147.14  WireGuard          1482       Transport Data, receiver=0xD8778C02, counter=11, datalen=1408
23           0.081532164       172.16.21.250     192.168.147.14  WireGuard          1482       Transport Data, receiver=0xD8778C02, counter=12, datalen=1408
24           0.081535164       172.16.21.250     192.168.147.14  WireGuard          1482       Transport Data, receiver=0xD8778C02, counter=13, datalen=1408
25           0.081538164       172.16.21.250     192.168.147.14  WireGuard          1482       Transport Data, receiver=0xD8778C02, counter=14, datalen=1408
26           0.081541064       172.16.21.250     192.168.147.14  WireGuard          410         Transport Data, receiver=0xD8778C02, counter=15, datalen=336

Cheers,

Sander Saares
Advisor

Axinom | Soola 8 | 51004 Tartu | Estonia
phone: +49 911 80109-54
saares at axinom.com<mailto:saares at axinom.com> | www.axinom.com<http://www.axinom.com/>

Managing Directors: Sergei Gussev, Oleg Knut
Tartu Circuit Court, Reg. 11046287

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.zx2c4.com/pipermail/wireguard/attachments/20191127/cfd37040/attachment-0001.html>