passing-through TOS/DSCP marking

[Toke Høiland-Jørgensen]

Daniel Golle <daniel at makrotopia.org> writes:

> On Fri, Jun 18, 2021 at 02:24:29PM +0200, Jason A. Donenfeld wrote:
>> Hey Toke,
>> 
>> On Fri, Jun 18, 2021 at 1:05 AM Toke Høiland-Jørgensen <toke at toke.dk> wrote:
>> > > I think you can achieve something similar using BPF filters, by relying
>> > > on wireguard passing through the skb->hash value when encrypting.
>> > >
>> > > Simply attach a TC-BPF filter to the wireguard netdev, pull out the DSCP
>> > > value and store it in a map keyed on skb->hash. Then, run a second BPF
>> > > filter on the physical interface that shares that same map, lookup the
>> > > DSCP value based on the skb->hash value, and rewrite the outer IP
>> > > header.
>> > >
>> > > The read-side filter will need to use bpf_get_hash_recalc() to make sure
>> > > the hash is calculated before the packet gets handed to wireguard, and
>> > > it'll be subject to hash collisions, but I think it should generally
>> > > work fairly well (for anything that's flow-based of course). And it can
>> > > be done without patching wireguard itself :)
>> >
>> > Just for fun I implemented such a pair of eBPF filters, and tested that
>> > it does indeed work for preserving DSCP marks on a Wireguard tunnel. The
>> > PoC is here:
>> >
>> > https://github.com/xdp-project/bpf-examples/tree/master/preserve-dscp
>> >
>> > To try it out (you'll need a recent-ish kernel and clang version) run:
>> >
>> > git clone --recurse-submodules https://github.com/xdp-project/bpf-examples
>> > cd bpf-examples/preserve-dscp
>> > make
>> > ./preserve-dscp wg0 eth0
>> >
>> > (assuming wg0 and eth0 are the wireguard and physical interfaces in
>> > question, respectively).
>> >
>> > To actually deploy this it would probably need a few tweaks; in
>> > particular the second filter that rewrites packets should probably check
>> > that the packets are actually part of the Wireguard tunnel in question
>> > (by parsing the UDP header and checking the source port) before writing
>> > anything to the packet.
>> >
>> > -Toke
>> 
>> That is a super cool approach. Thanks for writing that! Sounds like a
>> good approach, and one pretty easy to deploy, without the need to
>> patch kernels and such.
>> 
>> Also, nice usage of BPF_MAP_TYPE_LRU_HASH for this.
>> 
>> Daniel -- can you let the list know if this works for your use case?
>
> Turns out not exactly easy to deploy (on OpenWrt), as it depends on an
> extremely recent environment. I will try pushing to that direction, but
> it doesn't look like it's going to be ready very soon.
>
> In terms of toolchain: LLVM/Clang is a very bulky beast, I gave up on
> that and started working on integrating GCC-10's BPF target in our build
> system...

I saw that, but I have no idea if GCC's BPF target support will support
this. My tentative guess would be no, unfortunately :(

An alternative to getting LLVM built as part of the OpenWrt toolchain is
to just use the host clang to build the BPF binaries. It doesn't
actually need to be cross-compiled with a special compiler, the BPF byte
code format is the same on all architectures except for endianness, so
just passing that to the host clang should theoretically be enough...

> In terms of kernel support: recent kernels don't build yet because of
> gelf_getsymshndx, so we got to update libelf first for that. Recent
> libelf doesn't seem to be an option yet on many of the build hosts we
> currently support (Darwin and such).
>
> In terms of library support: our build of libbpf comes from Linux
> release tarballs. There isn't yet a release supporting bpf_tc_attach,
> the easiest would be to wait for Linux 5.13 to be released.

I used the libbpf TC loading support for convenience, but it's possible
to load it using 'tc' as well without too much trouble (right now the
userspace component sets a config variable before loading the program,
but it can be restructured to not need that).

Alternatively, the bpf-examples repository is setup with a libbpf
submodule that it can link statically against, so you could use that for
now?

> I (of course ;) also tried and spend almost a day looking for a
> quick-and-dirty path for temporary deployment, so I could at least give
> feedback -- bpf-examples also isn't exactly made to be cross-compiled
> manually, so I have failed with that as well so far.

Heh, no, it isn't, really. Anything in particular you need to make this
easier? We already added some bits to xdp-tools for supporting
cross-compilation (and that shares some lineage with bpf-examples), so
porting those over should not be too difficult.

See: https://github.com/xdp-project/xdp-tools/pull/78 and
https://github.com/xdp-project/xdp-tools/issues/74

Unfortunately I don't have a lot of time to poke more at this right now,
but feel free to open up an issue / pull request to the bpf-examples
repository with any changes you need :)

-Toke