<div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr">Ivan,<div><br></div><div>I tried the SNAT idea, and still have issue. </div><div>here is an example configuration of one of the nodes:</div><div><br></div><div><div>[Interface]</div><div>ListenPort = 5555 </div><div>PrivateKey = ---</div><div> </div><div>[Peer]</div><div>PublicKey = H09cwQeUUly2AIdTAhyr5zvzFK9bED0NYiKgJultYwE=</div><div>AllowedIPs = <a href="http://10.128.2.0/23">10.128.2.0/23</a></div><div>Endpoint = <a href="http://192.168.99.12:31112">192.168.99.12:31112</a></div><div>PersistentKeepalive = 25</div><div><br></div><div>[Peer]</div><div>PublicKey = 5nC5cyDg9WZ/2R3CPEbM+fSXzsn5Yx1mX48iizdfdHU=</div><div>AllowedIPs = <a href="http://10.128.0.0/23">10.128.0.0/23</a></div><div>Endpoint = <a href="http://192.168.99.14:32188">192.168.99.14:32188</a></div><div>PersistentKeepalive = 25</div><div><br></div><div>[Peer]</div><div>PublicKey = MzFg1tMaLUFC3kD9maiZZAHWywfCDyPlYF1zu6Dj30E=</div><div>AllowedIPs = <a href="http://10.130.0.0/23">10.130.0.0/23</a></div><div>Endpoint = <a href="http://192.168.99.13:31992">192.168.99.13:31992</a></div><div>PersistentKeepalive = 25</div><div><br></div><div>[Peer]</div><div>PublicKey = s7/lxyvFQCxE+KBoUJ/9vpPgLZ6pTdYUCsJ/snp3mUk=</div><div>AllowedIPs = <a href="http://10.129.0.0/23">10.129.0.0/23</a></div><div>Endpoint = <a href="http://192.168.99.15:30305">192.168.99.15:30305</a></div><div>PersistentKeepalive = 25</div><div><br></div><div>[Peer]</div><div>PublicKey = SuO927DbGm2h2I8hcf24LvYWglKp+4wGAuiyisin/yY=</div><div>AllowedIPs = <a href="http://10.131.0.0/23">10.131.0.0/23</a></div><div>Endpoint = <a href="http://192.168.99.7:31714">192.168.99.7:31714</a></div><div>PersistentKeepalive = 25</div><div><br></div><div>[Peer]</div><div>PublicKey = a+tK21LKdsBkQNqmqdRpvS9HLpz2W8rwDijTPkXEc0Q=</div><div>AllowedIPs = <a href="http://10.129.2.0/23">10.129.2.0/23</a></div><div>Endpoint = <a href="http://192.168.99.6:31165">192.168.99.6:31165</a></div><div>PersistentKeepalive = 25</div></div><div><br></div><div><font face="arial, helvetica, sans-serif">the private IP to VIP map for the peers of this node is: </font></div><div><br></div><div><span style="color:rgb(54,54,54);font-family:Menlo,Monaco,Consolas,monospace;font-size:12px;white-space:pre;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline">10.128.2.10-192.168.99.12
10.128.1.94-192.168.99.14
10.130.0.136-192.168.99.13
10.129.1.158-192.168.99.15
10.131.0.199-192.168.99.7
10.129.2.217-192.168.99.6 </span><br></div><div><span style="color:rgb(54,54,54);font-family:Menlo,Monaco,Consolas,monospace;font-size:12px;white-space:pre;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline"><br></span></div><div><span style="color:rgb(54,54,54);font-size:12px;white-space:pre;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline"><font face="arial, helvetica, sans-serif">I create the following iptables rules:</font></span></div><div><span style="color:rgb(54,54,54);font-size:12px;white-space:pre;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline"><font face="arial, helvetica, sans-serif"><br></font></span></div><div><span style="background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline"><font color="#363636" face="Menlo, Monaco, Consolas, monospace"><span style="font-size:12px;white-space:pre">sh-4.2# iptables -t nat -n -L
Chain PREROUTING (policy ACCEPT)
target prot opt source destination
Chain INPUT (policy ACCEPT)
target prot opt source destination
SNAT udp -- 10.128.2.10 <a href="http://0.0.0.0/0">0.0.0.0/0</a> udp dpt:5555 to:<a href="http://192.168.99.12:5555">192.168.99.12:5555</a>
SNAT udp -- 10.128.1.94 <a href="http://0.0.0.0/0">0.0.0.0/0</a> udp dpt:5555 to:<a href="http://192.168.99.14:5555">192.168.99.14:5555</a>
SNAT udp -- 10.130.0.136 <a href="http://0.0.0.0/0">0.0.0.0/0</a> udp dpt:5555 to:<a href="http://192.168.99.13:5555">192.168.99.13:5555</a>
SNAT udp -- 10.129.1.158 <a href="http://0.0.0.0/0">0.0.0.0/0</a> udp dpt:5555 to:<a href="http://192.168.99.15:5555">192.168.99.15:5555</a>
SNAT udp -- 10.131.0.199 <a href="http://0.0.0.0/0">0.0.0.0/0</a> udp dpt:5555 to:<a href="http://192.168.99.7:5555">192.168.99.7:5555</a>
SNAT udp -- 10.129.2.217 <a href="http://0.0.0.0/0">0.0.0.0/0</a> udp dpt:5555 to:<a href="http://192.168.99.6:5555">192.168.99.6:5555</a>
Chain OUTPUT (policy ACCEPT)
target prot opt source destination
Chain POSTROUTING (policy ACCEPT)
target prot opt source destination</span></font><br></span></div><div><span style="background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline"><font color="#363636" face="Menlo, Monaco, Consolas, monospace"><span style="font-size:12px;white-space:pre"><br></span></font></span></div><div><span style="background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline"><font color="#363636" face="Menlo, Monaco, Consolas, monospace"><span style="font-size:12px;white-space:pre"><br></span></font></span></div><div><span style="background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline"><font color="#363636" face="arial, helvetica, sans-serif"><span style="font-size:12px;white-space:pre">after the handshake the configuration changes to:</span></font></span></div><div><span style="background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline"><font color="#363636"><div style="font-family:Menlo,Monaco,Consolas,monospace;font-size:12px;white-space:pre"><br></div><div style="font-family:Menlo,Monaco,Consolas,monospace"><span style="font-size:12px;white-space:pre">sh-4.2# wg</span><br></div><div style="font-family:Menlo,Monaco,Consolas,monospace;font-size:12px;white-space:pre">interface: sdn-tunnel</div><div style="font-family:Menlo,Monaco,Consolas,monospace;font-size:12px;white-space:pre"> public key: gCFgNjOpObU71Vmjub/R9KIn3MHgzXnKtrh9Tf+W628=</div><div style="font-family:Menlo,Monaco,Consolas,monospace;font-size:12px;white-space:pre"> private key: (hidden)</div><div style="font-family:Menlo,Monaco,Consolas,monospace;font-size:12px;white-space:pre"> listening port: 5555</div><div style="font-family:Menlo,Monaco,Consolas,monospace;font-size:12px;white-space:pre"><br></div><div style="font-family:Menlo,Monaco,Consolas,monospace;font-size:12px;white-space:pre">peer: s7/lxyvFQCxE+KBoUJ/9vpPgLZ6pTdYUCsJ/snp3mUk=</div><div style="font-family:Menlo,Monaco,Consolas,monospace;font-size:12px;white-space:pre"> endpoint: <a href="http://10.134.0.1:1033">10.134.0.1:1033</a></div><div style="font-family:Menlo,Monaco,Consolas,monospace;font-size:12px;white-space:pre"> allowed ips: <a href="http://10.129.0.0/23">10.129.0.0/23</a></div><div style="font-family:Menlo,Monaco,Consolas,monospace;font-size:12px;white-space:pre"> latest handshake: 31 seconds ago</div><div style="font-family:Menlo,Monaco,Consolas,monospace;font-size:12px;white-space:pre"> transfer: 180 B received, 452 B sent</div><div style="font-family:Menlo,Monaco,Consolas,monospace;font-size:12px;white-space:pre"> persistent keepalive: every 25 seconds</div><div style="font-family:Menlo,Monaco,Consolas,monospace;font-size:12px;white-space:pre"><br></div><div style="font-family:Menlo,Monaco,Consolas,monospace;font-size:12px;white-space:pre">peer: MzFg1tMaLUFC3kD9maiZZAHWywfCDyPlYF1zu6Dj30E=</div><div style="font-family:Menlo,Monaco,Consolas,monospace;font-size:12px;white-space:pre"> endpoint: <a href="http://10.134.0.1:1032">10.134.0.1:1032</a></div><div style="font-family:Menlo,Monaco,Consolas,monospace;font-size:12px;white-space:pre"> allowed ips: <a href="http://10.130.0.0/23">10.130.0.0/23</a></div><div style="font-family:Menlo,Monaco,Consolas,monospace;font-size:12px;white-space:pre"> latest handshake: 37 seconds ago</div><div style="font-family:Menlo,Monaco,Consolas,monospace;font-size:12px;white-space:pre"> transfer: 212 B received, 272 B sent</div><div style="font-family:Menlo,Monaco,Consolas,monospace;font-size:12px;white-space:pre"> persistent keepalive: every 25 seconds</div><div style="font-family:Menlo,Monaco,Consolas,monospace;font-size:12px;white-space:pre"><br></div><div style="font-family:Menlo,Monaco,Consolas,monospace;font-size:12px;white-space:pre">peer: 5nC5cyDg9WZ/2R3CPEbM+fSXzsn5Yx1mX48iizdfdHU=</div><div style="font-family:Menlo,Monaco,Consolas,monospace;font-size:12px;white-space:pre"> endpoint: <a href="http://10.134.0.1:1031">10.134.0.1:1031</a></div><div style="font-family:Menlo,Monaco,Consolas,monospace;font-size:12px;white-space:pre"> allowed ips: <a href="http://10.128.0.0/23">10.128.0.0/23</a></div><div style="font-family:Menlo,Monaco,Consolas,monospace;font-size:12px;white-space:pre"> latest handshake: 39 seconds ago</div><div style="font-family:Menlo,Monaco,Consolas,monospace;font-size:12px;white-space:pre"> transfer: 180 B received, 304 B sent</div><div style="font-family:Menlo,Monaco,Consolas,monospace;font-size:12px;white-space:pre"> persistent keepalive: every 25 seconds</div><div style="font-family:Menlo,Monaco,Consolas,monospace;font-size:12px;white-space:pre"><br></div><div style="font-family:Menlo,Monaco,Consolas,monospace;font-size:12px;white-space:pre">peer: a+tK21LKdsBkQNqmqdRpvS9HLpz2W8rwDijTPkXEc0Q=</div><div style="font-family:Menlo,Monaco,Consolas,monospace;font-size:12px;white-space:pre"> endpoint: <a href="http://192.168.99.6:31165">192.168.99.6:31165</a></div><div style="font-family:Menlo,Monaco,Consolas,monospace;font-size:12px;white-space:pre"> allowed ips: <a href="http://10.129.2.0/23">10.129.2.0/23</a></div><div style="font-family:Menlo,Monaco,Consolas,monospace;font-size:12px;white-space:pre"> latest handshake: 41 seconds ago</div><div style="font-family:Menlo,Monaco,Consolas,monospace;font-size:12px;white-space:pre"> transfer: 156 B received, 180 B sent</div><div style="font-family:Menlo,Monaco,Consolas,monospace;font-size:12px;white-space:pre"> persistent keepalive: every 25 seconds</div><div style="font-family:Menlo,Monaco,Consolas,monospace;font-size:12px;white-space:pre"><br></div><div style="font-family:Menlo,Monaco,Consolas,monospace;font-size:12px;white-space:pre">peer: H09cwQeUUly2AIdTAhyr5zvzFK9bED0NYiKgJultYwE=</div><div style="font-family:Menlo,Monaco,Consolas,monospace;font-size:12px;white-space:pre"> endpoint: <a href="http://192.168.99.12:31112">192.168.99.12:31112</a></div><div style="font-family:Menlo,Monaco,Consolas,monospace;font-size:12px;white-space:pre"> allowed ips: <a href="http://10.128.2.0/23">10.128.2.0/23</a></div><div style="font-family:Menlo,Monaco,Consolas,monospace;font-size:12px;white-space:pre"> latest handshake: 41 seconds ago</div><div style="font-family:Menlo,Monaco,Consolas,monospace;font-size:12px;white-space:pre"> transfer: 156 B received, 180 B sent</div><div style="font-family:Menlo,Monaco,Consolas,monospace;font-size:12px;white-space:pre"> persistent keepalive: every 25 seconds</div><div style="font-family:Menlo,Monaco,Consolas,monospace;font-size:12px;white-space:pre"><br></div><div style="font-family:Menlo,Monaco,Consolas,monospace;font-size:12px;white-space:pre">peer: SuO927DbGm2h2I8hcf24LvYWglKp+4wGAuiyisin/yY=</div><div style="font-family:Menlo,Monaco,Consolas,monospace;font-size:12px;white-space:pre"> endpoint: <a href="http://192.168.99.7:31714">192.168.99.7:31714</a></div><div style="font-family:Menlo,Monaco,Consolas,monospace;font-size:12px;white-space:pre"> allowed ips: <a href="http://10.131.0.0/23">10.131.0.0/23</a></div><div style="font-family:Menlo,Monaco,Consolas,monospace;font-size:12px;white-space:pre"> latest handshake: 41 seconds ago</div><div style="font-family:Menlo,Monaco,Consolas,monospace;font-size:12px;white-space:pre"> transfer: 156 B received, 180 B sent</div><div style="font-family:Menlo,Monaco,Consolas,monospace;font-size:12px;white-space:pre"> persistent keepalive: every 25 seconds</div><div style="font-family:Menlo,Monaco,Consolas,monospace;font-size:12px;white-space:pre"><br></div><div style="font-size:12px;white-space:pre"><font face="arial, helvetica, sans-serif">as you can see some of the endpoint's addresses have changed. </font></div><div style="font-size:12px;white-space:pre"><font face="arial, helvetica, sans-serif">the first three are not correct anymore.</font></div><div style="font-size:12px;white-space:pre"><font face="arial, helvetica, sans-serif">After the introduction of the iptables rules they change to an IP that makes no sense to me <span style="font-family:Menlo,Monaco,Consolas,monospace;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline">10.134.0.1</span></font></div><div style="font-size:12px;white-space:pre"><font face="arial, helvetica, sans-serif"><span style="font-family:Menlo,Monaco,Consolas,monospace;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline"><br></span></font></div><div style="font-size:12px;white-space:pre"><font face="arial, helvetica, sans-serif"><span style="background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline">finally here are a few seconds of tcpdump in case it helps:</span></font></div><div style="font-size:12px;white-space:pre"><font face="arial, helvetica, sans-serif"><span style="font-family:Menlo,Monaco,Consolas,monospace;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline"><br></span></font></div><div><font><span style="background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline"><font face="Menlo, Monaco, Consolas, monospace"><span style="font-size:12px;white-space:pre">sh-4.2# tcpdump -i eth0 -nn -v
tcpdump: listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes
23:07:01.045331 IP (tos 0x0, ttl 64, id 27711, offset 0, flags [none], proto UDP (17), length 60)
10.134.0.72.5555 > 192.168.99.6.31165: UDP, length 32
23:07:01.045363 IP (tos 0x0, ttl 64, id 53835, offset 0, flags [none], proto UDP (17), length 60)
10.134.0.72.5555 > 192.168.99.7.31714: UDP, length 32
23:07:01.045411 IP (tos 0x0, ttl 64, id 27009, offset 0, flags [none], proto UDP (17), length 60)
10.134.0.72.5555 > 192.168.99.12.31112: UDP, length 32
23:07:02.758694 IP (tos 0x0, ttl 61, id 19309, offset 0, flags [none], proto UDP (17), length 60)
10.134.0.1.1031 > 10.134.0.72.5555: UDP, length 32
23:07:04.053339 IP (tos 0x0, ttl 64, id 36786, offset 0, flags [none], proto UDP (17), length 60)
10.134.0.72.5555 > 10.134.0.1.1032: UDP, length 32
23:07:07.765375 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 10.134.0.72 tell 10.134.0.1, length 28
23:07:07.765394 ARP, Ethernet (len 6), IPv4 (len 4), Reply 10.134.0.72 is-at 0a:58:0a:86:00:48, length 28
23:07:10.938921 IP (tos 0x0, ttl 61, id 33093, offset 0, flags [none], proto UDP (17), length 60)
10.134.0.1.1033 > 10.134.0.72.5555: UDP, length 32
23:07:26.069271 IP (tos 0x0, ttl 64, id 37778, offset 0, flags [none], proto UDP (17), length 60)
10.134.0.72.5555 > 192.168.99.6.31165: UDP, length 32
23:07:26.069271 IP (tos 0x0, ttl 64, id 59175, offset 0, flags [none], proto UDP (17), length 60)
10.134.0.72.5555 > 192.168.99.7.31714: UDP, length 32
23:07:26.069303 IP (tos 0x0, ttl 64, id 49067, offset 0, flags [none], proto UDP (17), length 60)
10.134.0.72.5555 > 192.168.99.12.31112: UDP, length 32
23:07:27.797284 IP (tos 0x0, ttl 64, id 57007, offset 0, flags [none], proto UDP (17), length 60)
10.134.0.72.5555 > 10.134.0.1.1031: UDP, length 32
23:07:29.079935 IP (tos 0x0, ttl 61, id 18743, offset 0, flags [none], proto UDP (17), length 60)
10.134.0.1.1032 > 10.134.0.72.5555: UDP, length 32</span></font><br></span></font></div></font></span></div></div></div></div></div></div></div><div class="gmail_extra"><br clear="all"><div><div class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr">Thanks,<div>Raffaele</div><div><br></div><div>Raffaele Spazzoli</div><div>Senior Architect - <a href="https://www.openshift.com" target="_blank">OpenShift</a>, <a href="https://www.redhat.com/en/services/consulting/paas" target="_blank">Containers and PaaS Practice</a></div><div>Tel: +1 216-258-7717</div><div><br></div><div><br></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div>
<br><div class="gmail_quote">On Sun, Sep 16, 2018 at 2:56 PM, Raffaele Spazzoli <span dir="ltr"><<a href="mailto:rspazzol@redhat.com" target="_blank">rspazzol@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">I'll try to make an example<div>cluster 1 node 1 has private IP1 and VIP1 </div><div>cluster 2 node 2 has private IP2 and VIP2</div><div class="gmail_extra"><br></div><div class="gmail_extra">each node uses it's private ip for outbound connections. </div><div class="gmail_extra">each node can receive inbound connection on its VIP.</div><div class="gmail_extra"><br></div><div class="gmail_extra">so the wireguard config file for node1 is going to look like:</div><div class="gmail_extra"><br></div><div class="gmail_extra">[peer]</div><div class="gmail_extra">endpoint: VIP2:port</div><div class="gmail_extra"><br></div><div class="gmail_extra">and for node 2:</div><div class="gmail_extra">[peer] </div><div class="gmail_extra">endpoint: VIP1: port</div><div class="gmail_extra"><br></div><div class="gmail_extra">the problem is that after the handshake, wireguard updates the config to the following (for example for node2):</div><div class="gmail_extra">[peer]</div><div class="gmail_extra">endpoint: IP1:port</div><div class="gmail_extra"><br></div><div class="gmail_extra">but IP2 cannot route to IP1...</div><div class="gmail_extra"><br></div><div class="gmail_extra">I think a well configured SNAT rule may work, although is not elegant because it forces the cluster to exchange information about their private IPs.</div><div class="gmail_extra">This should not be needed and in the cloud private IPs are ephemeral....</div><div class="gmail_extra"><br></div><div class="gmail_extra">anyway thanks for the advice, I am going to try to use it in my prototype. </div><div class="gmail_extra"><br></div><div class="gmail_extra">I still think there is need for a better technical approach for a long term solution.</div><div class="gmail_extra"><br></div><div class="gmail_extra"><span class=""><br clear="all"><div><div class="m_4340499724176942754gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr">Thanks,<div>Raffaele</div><div><br></div><div>Raffaele Spazzoli</div><div>Senior Architect - <a href="https://www.openshift.com" target="_blank">OpenShift</a>, <a href="https://www.redhat.com/en/services/consulting/paas" target="_blank">Containers and PaaS Practice</a></div><div>Tel: +1 216-258-7717</div><div><br></div><div><br></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div>
<br></span><div><div class="h5"><div class="gmail_quote">On Sun, Sep 16, 2018 at 12:54 PM, Ivan Labáth <span dir="ltr"><<a href="mailto:labawi-wg@matrix-dream.net" target="_blank">labawi-wg@matrix-dream.net</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hi,<br>
<br>
On Sun, Sep 16, 2018 at 08:21:02AM -0400, Raffaele Spazzoli wrote:<br>
> ... then the IP that a node uses for its outbound<br>
<span>> connection is not the same that its peer need to use for its inbound<br>
> connections.<br>
<br>
</span>Who uses what for whose connection? You lost me here.<br>
Looks like a broken network to me. Does TCP even work?<br>
<br>
Anyway, SNAT/DNAT should be able to fix things up, if you want to go<br>
that route.<br>
<br>
Regards,<br>
Ivan<br>
</blockquote></div><br></div></div></div></div>
</blockquote></div><br></div>