T-Mobile 4G/5G CGNAT vs WireGuard tunnel jitter

[Lonnie Abelbeck]

Hi Mo,

I have the latest hardware (gray cylinder)
--
Typically 3 of 5 bars
4G Band: B2
5G Band: n71
--

Note the huge jitter only occurs inbound, which is difficult to measure without using WireGuard.

The same UDP iperf3 tests outbound (gw-lan->linode) range from 3 to 8 ms jitter, regardless of bitrate.

--
Speed ( less than 2 seconds) 10/15 Mbps (down/up)
slowly ramps up to ...
Speed ( after 20 seconds) up to 200/35 Mbps (down/up)
--

Lonnie


> On Apr 10, 2021, at 10:43 AM, Mo Balaa <buddybalaa at gmail.com> wrote:
> 
> Thanks for sharing, I have also been running WG tunnels over T-Mobile home internet and haven’t seen any of the jitter you are reporting. 
> 
> Did you try the same tests (outbound) without running them via WG?
> Which modem do you have? How many signal bars are you getting? Also, what does an non-tunneled speed test report?
> 
> Cheers 
> 
> 
>> On Apr 10, 2021, at 10:31, Lonnie Abelbeck <lists at lonnie.abelbeck.com> wrote:
>> 
>> Greetings,
>> 
>> I have been testing the T-Mobile Home Internet (4G/5G fixed wireless) service to a Linode VM via WireGuard.
>> 
>> The TMHI service uses CGNAT plus an additional NAT in their modem/gateway with a MTU of 1420, so WireGuard is configured with a 1340 MTU.
>> 
>> Everything works, but I thought I would share some jitter results that readers here might find interesting.
>> 
>> [gw-lan WGIP:10.4.1.1] -- [TMHI modem/gateway] -- 4G/5G/CGNAT -- [linode WGIP:10.4.1.10]
>> 
>> gw-lan ~ # mtr -wn -c 30 -s 1340 10.4.1.10
>> ...
>> HOST: gw-lan      Loss%   Snt   Last   Avg  Best  Wrst StDev
>> 1.|-- 10.4.1.10  0.0%    30   88.7  88.9  77.2  99.2   5.4
>> 
>> Looks to be as expected, in the direction of the CGNAT, now the other direction, against the grain of the CGNAT ...
>> 
>> linode ~ # mtr -wn -c 30 -s 1340 10.4.1.1
>> ...
>> HOST: linode     Loss%   Snt   Last   Avg  Best  Wrst StDev
>> 1.|-- 10.4.1.1  0.0%    30  206.1 243.5  73.8 393.9  97.9
>> 
>> Huge jitter, and is very reproducible.  But no packet loss.
>> 
>> Further investigation shows for low traffic rates (linode->gw-lan) the jitter over WireGuard is huge, here are some UDP iperf3 tests showing how the jitter goes down as the traffic rate is increased.
>> 
>> linode ~ # iperf3 -c 10.4.1.1 -u -b 5k -t 30
>> ...
>> [ ID] Interval           Transfer     Bitrate         Jitter    Lost/Total Datagrams
>> [  5]   0.00-30.25  sec  18.9 KBytes  5.11 Kbits/sec  68.428 ms  0/15 (0%)  receiver
>> 
>> linode ~ # iperf3 -c 10.4.1.1 -u -b 10k -t 30
>> ...
>> [ ID] Interval           Transfer     Bitrate         Jitter    Lost/Total Datagrams
>> [  5]   0.00-30.30  sec  37.7 KBytes  10.2 Kbits/sec  82.411 ms  0/30 (0%)  receiver
>> 
>> linode ~ # iperf3 -c 10.4.1.1 -u -b 50k -t 30
>> ...
>> [ ID] Interval           Transfer     Bitrate         Jitter    Lost/Total Datagrams
>> [  5]   0.00-30.14  sec   184 KBytes  49.9 Kbits/sec  7.532 ms  0/146 (0%)  receiver
>> 
>> linode ~ # iperf3 -c 10.4.1.1 -u -b 100k -t 30
>> ...
>> [ ID] Interval           Transfer     Bitrate         Jitter    Lost/Total Datagrams
>> [  5]   0.00-30.10  sec   367 KBytes   100 Kbits/sec  4.182 ms  0/292 (0%)  receiver
>> 
>> linode ~ # iperf3 -c 10.4.1.1 -u -b 500k -t 30
>> ...
>> [ ID] Interval           Transfer     Bitrate         Jitter    Lost/Total Datagrams
>> [  5]   0.00-30.11  sec  1.79 MBytes   498 Kbits/sec  1.308 ms  0/1456 (0%)  receiver
>> 
>> 
>> So using VoIP a higher bitrate CODEC is actually better w.r.t jitter.
>> 
>> Hope others find this interesting.
>> 
>> Lonnie
>> 
> 
>