Connection hangs over CGNAT (Starlink)

[reox]

Interesting, because I was going to post a similar question here - but 
would not have thought about multi-network.
For me, this happens on my Android phone, where I use WG to route DNS 
traffic to my own server.
In some wifi networks, I get this issue that after some time (sometimes 
only minutes, sometimes hours), that for example K9mail is unable to 
fetch mails because it presumably runs into a DNS timeout (it cannot be 
a connection timeout to the mail server, because that is not routed via WG)
Toggling Wifi, switching to mobile network only, or toggling wireguard 
solves the issue.
I had this problem, however, also rarely in other wifis. Before that, I 
thought that the wifi network itself was the culprit, but as it happens 
also on other occasions, thus I thought it might be a combination of 
specific wifi setup and my server setup.
However, I have no idea how I could debug this, especially as DNS 
requests using termux and dig work flawlessly, even though at the same 
time k9mail hangs.
Using KeepAlive did not work so far.
I wanted to debug this further by running tcpdump on the server, but 
unfortunately, I have right now no wifi where I can trigger this bug 
reliably. In my own wifi, it happens every now and then - typically only 
once a month or less...

Best,
Sebastian

On 17.12.2022 23:15, Szymon Nowak wrote:
> I've noticed the same thing on a WIndows client, it happens when you
> provide internet from two sources, e.g. Wifi and mobile network or
> Wifi and LAN in case of computer, When one of these sources has a
> problem and internet is not available on it. Then Wireguard stops
> working even though it doesn't break the tunnel. Completely
> disconnecting the faulty connection and reconnecting the tunleu solves
> this problem. I don't know why they don't work even though the tunnel
> is connected
> 
> On Sat, Dec 17, 2022 at 11:05 PM Nikolay Martynov <mar.kolya at gmail.com> wrote:
>>
>> Hi!
>>
>> I'm experiencing strange behaviour with wireguard: from time to time
>> connection 'freezes'.
>> Most often I'm observing this on an Android phone when connected from
>> my home over Starlink.
>> Server: latest Openwrt, Client: latest Android app.
>> The connection establishes and works fine for some time. After some
>> time the client still shows connection is established, but no incoming
>> data is coming.
>> On a server side 'latest handshake' goes into hours/days.
>> The freeze happens randomly, for no apparent reason and I think only
>> over starlink. I do not think I have ever observed this problem on
>> cell networks.
>>
>> Reconnection solves the problem immediately.
>> I did some tcpdumping when the problem was present and found the following:
>> * Server side sees incoming traffic from the client and sends responses.
>> * On my own router connected to Starlink (i.e. interface between my
>> router and Starlink router) I see data going from the client to the
>> server - but no packets coming back.
>>
>> So my 'hypothesis' is that somehow Starlink's CGNAT 'forgets' one side
>> of the connection - and so data continues to go in one direction, but
>> it doesn't come back. The thing with the wireguard is that it looks
>> like it doesn't change the outgoing port when it attempts to do
>> another handshake. This means that it continues using the same 'half
>> broken' connection forever.
>>
>> I think the same happens to me at least once on a Linux client - but
>> the difference with the phone is that the phone is always on and
>> therefore the duration of the connection is much longer.
>>
>> I tried experimenting with keepalive messages - but it looks like they
>> make no difference. Once connection freezes I see keepalived arriving
>> onto the server, server sending reply - but that reply never arrives
>> to the client.
>>
>> It looks like the solution to this problem would be for the client to
>> use a different outgoing port when sending a handshake but I was not
>> able to find an option for that.
>>
>> Is this something that is possible to do?
>> Thanks!
>>
>>
>> --
>> Martynov Nikolay.
>> Email: mar.kolya at gmail.com