multiple endpoints for a single peer -- implementation details

[Motiejus Jakštys]

Hi all,

I thought I'd implement a prototype to use multiple endpoints for a
single peer, but after some analysis on "## Select new endpoint during
each handshake"[1], I'd like to share the concerns with future readers
who might try the same endeavor. TLDR: I think the kernel is not in
the best position to do this, "decision making in user space" may be
more appropriate.

To make it happen, handshake process would change. New suggested flow:
- Initiator sends a handshake packet to all endpoints quasi-simultaneously.
  - Each handshake is a new message with a different ephemeral key et al.
- Responder receives the first one and responds.
- Responder receives more handshakes within 1/INITIATIONS_PER_SECOND
and discards them.
- Responder may receive more after 1/INITIATIONS_PER_SECOND and responds.

Responder needs to maintain more than one handshake state for
MAX_TIMER_HANDSHAKES, specifically, the whole `noise_handshake`
struct. Following a later suggestion in the thread, this can have an
upper bound of MAX_ENDPOINTS_PER_PEER (TBD constant).

Responder's responses are technically different handshakes: different
ephemeral keys, different hashes, possibly different indices (I
haven't figured the role of indices yet). From this stems the question
1.

Questions:
1. how can initiator associate the one-of-the-many handshakes it sent
with the one-of-the-many responses it received? I.e. is there
common/derivable data shared between the handshake request and
response? I wasn't able to find one.
2. more a concern: neither kernel, nor wireguard-go implementations
are willing to accept more than one endpoint, and it would be messy to
extend:

include/uapi/linux/wireguard.h
  WGPEER_A_ENDPOINT: struct sockaddr_in or struct sockaddr_in6

device/uapi.go:
  case "endpoint":
  ...
    endpoint, err := CreateEndpoint(value)

Endpoint is fixed to be a single UDP address, and both kernel and
wireguard-go refuse unknown keys. To have tooling
backwards-compatibility (i.e. use newer wireguard-tools with older
kernel implementations), wireguard-tools would need to know the
supported "features" of the underlying implementation. And there is no
version negotiation between the user/kernel space. Which makes it
tricky to add features like this.

Related gotcha: currently DNS is looked up in userspace during
interface configuration**. The biggest selling point of "multiple
endpoints" is increasing probability to reach it... Given that:
1. wireguard-linux netlink API was not designed to be extendable (so
isn't wireguard-go).
2. DNS lookup is done on configuration time.

I am suggesting that "## Decision-making in userspace" would work
better here. Userspace would regularly* issue handshake initiations
and measure how long it takes for each endpoint, and hand over the
*single* endpoint to the kernel to connect. Interestingly enough, this
"userspace thing" is a combination of wireguard-tools for config
parsing and MNL, and wireguard-go to initiate the handshake. What do
you think, where could this belong? A userspace wireguard
implementation in C, and/or wireguard-tools implementation in Go would
make this easy, but we have neither. :)

Motiejus

[1]: https://lists.zx2c4.com/pipermail/wireguard/2017-January/000917.html
[*]: timing and conditions when to do the "probing handshakes" would
need to be very carefully considered. TBD.
[**]: my wireguard server is behind a dynamic IP with a DNS TTL of 600 seconds.