Transient Connection Issue

[Ashish Madeti]

Hi All

Background: I am using Wireguard VPN to secure intra-server
communications among my 5-6 ubuntu servers sitting in different data
centers.

Today, we had a downtime of around 15 minutes because the server
running nginx was not able to connect to the web-application server
using the wireguard interface [0]. I ascertained that it was not a
connection issue between nginx server and web-application server by
trying to connect to web-application server via its public IP, which
worked [1]. I even tried restarting wireguard service [2] on both
nginx and web-application server but to no avail.
So, before investigating further, I decided to first route all the
traffic to a failover server (which was also a part of the VPN). It
took me around 5-10 minutes to pull the latest configuration and
application changes onto the failover server and then route all
traffic to it. Once our site was up, I again tried connecting to the
original web-application server from nginx server, using curl, but
this time it worked fine.

Can anybody help me understand the problem or anything I should try if
it happens again?

Please let me know if you need any more information.

[0] Tried via curl. curl 10.0.0.10:8080. Received the error
'Connection timed out'
[1] curl w.x.y.z:8080 returned the html content as expected.
[2] sudo service wg-quick at wg0 restart

Regards
-- 
Ashish Madeti