<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:Wingdings;
panose-1:5 0 0 0 0 0 0 0 0 0;}
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:#0563C1;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:#954F72;
text-decoration:underline;}
p.MsoListParagraph, li.MsoListParagraph, div.MsoListParagraph
{mso-style-priority:34;
margin-top:0cm;
margin-right:0cm;
margin-bottom:0cm;
margin-left:36.0pt;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
span.EmailStyle17
{mso-style-type:personal-compose;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri",sans-serif;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:72.0pt 72.0pt 72.0pt 72.0pt;}
div.WordSection1
{page:WordSection1;}
/* List Definitions */
@list l0
{mso-list-id:1170170703;
mso-list-type:hybrid;
mso-list-template-ids:-1522905622 1350069654 67698691 67698693 67698689 67698691 67698693 67698689 67698691 67698693;}
@list l0:level1
{mso-level-start-at:0;
mso-level-number-format:bullet;
mso-level-text:\F0B7;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Symbol;
mso-fareast-font-family:Calibri;
mso-bidi-font-family:Arial;}
@list l0:level2
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:"Courier New";}
@list l0:level3
{mso-level-number-format:bullet;
mso-level-text:\F0A7;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Wingdings;}
@list l0:level4
{mso-level-number-format:bullet;
mso-level-text:\F0B7;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Symbol;}
@list l0:level5
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:"Courier New";}
@list l0:level6
{mso-level-number-format:bullet;
mso-level-text:\F0A7;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Wingdings;}
@list l0:level7
{mso-level-number-format:bullet;
mso-level-text:\F0B7;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Symbol;}
@list l0:level8
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:"Courier New";}
@list l0:level9
{mso-level-number-format:bullet;
mso-level-text:\F0A7;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Wingdings;}
ol
{margin-bottom:0cm;}
ul
{margin-bottom:0cm;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="#0563C1" vlink="#954F72">
<div class="WordSection1">
<p class="MsoNormal">Hi!<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">I am troubleshooting some odd behavior and noticed that when my Ubuntu 18.04 server is routing packets between the internet and a WireGuard tunnel, there exist some odd differences between how different connections are handled that I cannot
account for.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">In both cases, I have a client (192.168.147.14) from the other side of the tunnel making an HTTPS connection. The IPv4 packets received from the internet have the “don’t fragment” flag set, as is common. What I find is:<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<ul style="margin-top:0cm" type="disc">
<li class="MsoListParagraph" style="margin-left:0cm;mso-list:l0 level1 lfo1">When processing large packets from server A (159.148.147.196), Ubuntu responds with “needs fragmentation” and correctly gives the WireGuard MTU of 1420 as the next hop MTU.<o:p></o:p></li><li class="MsoListParagraph" style="margin-left:0cm;mso-list:l0 level1 lfo1">When processing large packets from server B (40.68.232.16), Ubuntu just does the fragmentation itself and forwards the packets no questions asked. Despite “don’t fragment” bit being
set.<o:p></o:p></li></ul>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">I am unable to explain the difference in behavior. What am I missing here? Why might the two cases be handled differently by the operating system? I do not even know if this is WireGuard related – maybe it also occurs with non-WireGuard
adapters.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">My forwarding-relevant WireGuard configuration is simply the following:<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">PostUp = iptables -A FORWARD -i wg0 -j ACCEPT; iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE<o:p></o:p></p>
<p class="MsoNormal">PostDown = iptables -D FORWARD -i wg0 -j ACCEPT; iptables -t nat -D POSTROUTING -o eth0 -j MASQUERADE<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">I have not adjusted any relevant operating system settings from defaults other than allowing IP forwarding.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">The initial packets of the connection that do not exceed the WireGuard MTU are correctly routed through the WireGuard tunnel in both cases.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><b>Response from server A<o:p></o:p></b></p>
<p class="MsoNormal">Internet Protocol Version 4, Src: 159.148.147.196, Dst: 172.16.21.250<o:p></o:p></p>
<p class="MsoNormal"> 0100 .... = Version: 4<o:p></o:p></p>
<p class="MsoNormal"> .... 0101 = Header Length: 20 bytes (5)<o:p></o:p></p>
<p class="MsoNormal"> Differentiated Services Field: 0x00 (DSCP: CS0, ECN: Not-ECT)<o:p></o:p></p>
<p class="MsoNormal"> Total Length: 3042<o:p></o:p></p>
<p class="MsoNormal"> Identification: 0x2839 (10297)<o:p></o:p></p>
<p class="MsoNormal"><b> Flags: 0x4000, Don't fragment<o:p></o:p></b></p>
<p class="MsoNormal"> ...0 0000 0000 0000 = Fragment offset: 0<o:p></o:p></p>
<p class="MsoNormal"> Time to live: 46<o:p></o:p></p>
<p class="MsoNormal"> Protocol: TCP (6)<o:p></o:p></p>
<p class="MsoNormal"> Header checksum: 0x237a [validation disabled]<o:p></o:p></p>
<p class="MsoNormal"> [Header checksum status: Unverified]<o:p></o:p></p>
<p class="MsoNormal"> Source: 159.148.147.196<o:p></o:p></p>
<p class="MsoNormal"> Destination: 172.16.21.250<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><b>In response, “fragmentation needed” is sent<o:p></o:p></b></p>
<p class="MsoNormal">Internet Protocol Version 4, Src: 172.16.21.250, Dst: 159.148.147.196<o:p></o:p></p>
<p class="MsoNormal">Internet Control Message Protocol<o:p></o:p></p>
<p class="MsoNormal"> Type: 3 (Destination unreachable)<o:p></o:p></p>
<p class="MsoNormal"> Code: 4 (Fragmentation needed)<o:p></o:p></p>
<p class="MsoNormal"> Checksum: 0x99ba [correct]<o:p></o:p></p>
<p class="MsoNormal"> [Checksum Status: Good]<o:p></o:p></p>
<p class="MsoNormal"> Unused: a8<o:p></o:p></p>
<p class="MsoNormal"> Length: 246<o:p></o:p></p>
<p class="MsoNormal"> [Length of original datagram: 984]<o:p></o:p></p>
<p class="MsoNormal"> MTU of next hop: 1420<o:p></o:p></p>
<p class="MsoNormal"> Internet Protocol Version 4, Src: 159.148.147.196, Dst: 172.16.21.250<o:p></o:p></p>
<p class="MsoNormal"> Transmission Control Protocol, Src Port: 443, Dst Port: 43254, Seq: 2041044714, Ack: 1154907605<o:p></o:p></p>
<p class="MsoNormal"> Transport Layer Security<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><b>Response from server B</b><o:p></o:p></p>
<p class="MsoNormal">Internet Protocol Version 4, Src: 40.68.232.16, Dst: 172.16.21.250<o:p></o:p></p>
<p class="MsoNormal"> 0100 .... = Version: 4<o:p></o:p></p>
<p class="MsoNormal"> .... 0101 = Header Length: 20 bytes (5)<o:p></o:p></p>
<p class="MsoNormal"> Differentiated Services Field: 0x00 (DSCP: CS0, ECN: Not-ECT)<o:p></o:p></p>
<p class="MsoNormal"> Total Length: 5716<o:p></o:p></p>
<p class="MsoNormal"> Identification: 0x22e1 (8929)<o:p></o:p></p>
<p class="MsoNormal"><b> Flags: 0x4000, Don't fragment<o:p></o:p></b></p>
<p class="MsoNormal"> ...0 0000 0000 0000 = Fragment offset: 0<o:p></o:p></p>
<p class="MsoNormal"> Time to live: 119<o:p></o:p></p>
<p class="MsoNormal"> Protocol: TCP (6)<o:p></o:p></p>
<p class="MsoNormal"> Header checksum: 0xf863 [validation disabled]<o:p></o:p></p>
<p class="MsoNormal"> [Header checksum status: Unverified]<o:p></o:p></p>
<p class="MsoNormal"> Source: 40.68.232.16<o:p></o:p></p>
<p class="MsoNormal"> Destination: 172.16.21.250<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><b>In response, Ubuntu just fragments the packet anyway and pushes through the WG tunnel<o:p></o:p></b></p>
<p class="MsoNormal">22 0.081525663 172.16.21.250 192.168.147.14 WireGuard 1482 Transport Data, receiver=0xD8778C02, counter=11, datalen=1408<o:p></o:p></p>
<p class="MsoNormal">23 0.081532164 172.16.21.250 192.168.147.14 WireGuard 1482 Transport Data, receiver=0xD8778C02, counter=12, datalen=1408<o:p></o:p></p>
<p class="MsoNormal">24 0.081535164 172.16.21.250 192.168.147.14 WireGuard 1482 Transport Data, receiver=0xD8778C02, counter=13, datalen=1408<o:p></o:p></p>
<p class="MsoNormal">25 0.081538164 172.16.21.250 192.168.147.14 WireGuard 1482 Transport Data, receiver=0xD8778C02, counter=14, datalen=1408<o:p></o:p></p>
<p class="MsoNormal">26 0.081541064 172.16.21.250 192.168.147.14 WireGuard 410 Transport Data, receiver=0xD8778C02, counter=15, datalen=336<o:p></o:p></p>
<p class="MsoNormal" style="text-align:justify"><span lang="ET" style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal" style="text-align:justify"><span lang="ET" style="color:#1F497D">Cheers,<o:p></o:p></span></p>
<p class="MsoNormal" style="text-align:justify"><span lang="EN-GB" style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal" style="text-align:justify"><b><span lang="EN-GB" style="color:#002060">Sander Saares</span></b><span lang="EN-GB" style="font-size:14.0pt;color:#002060"><o:p></o:p></span></p>
<p class="MsoNormal" style="text-align:justify"><span style="font-size:10.0pt;color:#002060">Advisor
</span><span lang="EN-GB" style="font-size:14.0pt;color:#002060"><o:p></o:p></span></p>
<p class="MsoNormal" style="text-align:justify"><span style="font-size:14.0pt;color:#595959"> </span><span lang="EN-GB" style="font-size:14.0pt;color:#595959"><o:p></o:p></span></p>
<p class="MsoNormal"><b><span style="font-size:10.0pt;color:#7F7F7F">Axinom </span></b><span style="font-size:10.0pt;color:#7F7F7F">| Soola 8 | 51004 Tartu | Estonia</span><span lang="EN-GB" style="font-size:14.0pt;color:#7F7F7F"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;color:#7F7F7F">phone: +49 911 80109-54<o:p></o:p></span></p>
<p class="MsoNormal"><a href="mailto:saares@axinom.com"><span style="font-size:10.0pt;color:#0563C1">saares@axinom.com</span></a><span style="font-size:10.0pt;color:black">
</span><span style="font-size:10.0pt;color:#7F7F7F">| </span><a href="http://www.axinom.com/"><span style="font-size:10.0pt;color:#0563C1">www.axinom.com</span></a><span style="font-size:10.0pt;color:black">
</span><span lang="EN-GB" style="font-size:10.0pt;color:#7F7F7F"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;color:#7F7F7F"> </span><span lang="EN-GB" style="font-size:14.0pt;color:#7F7F7F"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:9.0pt;color:#7F7F7F">Managing Directors: Sergei Gussev, Oleg Knut<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:9.0pt;color:#7F7F7F">Tartu Circuit Court, Reg. 11046287</span><span lang="EN-GB" style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</body>
</html>