passing-through TOS/DSCP marking

Sun Jul 4 14:15:18 UTC 2021

Hi Toke,

thank you for the ongoing efforts and support on this issue.

On Wed, Jun 30, 2021 at 10:55:09PM +0200, Toke Høiland-Jørgensen wrote:
> Daniel Golle <daniel at makrotopia.org> writes:
> > ...
> >> >
> >> > In terms of toolchain: LLVM/Clang is a very bulky beast, I gave up on
> >> > that and started working on integrating GCC-10's BPF target in our build
> >> > system...
> >> 
> >> I saw that, but I have no idea if GCC's BPF target support will support
> >> this. My tentative guess would be no, unfortunately :(
> >
> > Probably you are right. When building the BPF object with GCC, the
> > result is:
> > root at OpenWrt:/usr/lib/bpf# preserve-dscp wg0 eth0
> > libbpf: elf: skipping unrecognized data section(4) .stab
> > libbpf: elf: skipping relo section(5) .rel.stab for section(4) .stab
> > libbpf: elf: skipping unrecognized data section(13) .comment
> > libbpf: BTF is required, but is missing or corrupted.
> > Couldn't open file: preserve_dscp_kern.o
> 
> Hmm, for this example it should be possible to make it run without BTF.
> I'm only using that for the map definition, so that could be changed to
> the old format; you could try this patch:
> 
> diff --git a/preserve-dscp/preserve_dscp_kern.c b/preserve-dscp/preserve_dscp_kern.c
> index 24120cb8a3ff..08248e1f0e41 100644
> --- a/preserve-dscp/preserve_dscp_kern.c
> +++ b/preserve-dscp/preserve_dscp_kern.c
> @@ -9,12 +9,12 @@
>   * otherwise clean up stale entries. Instead, we just rely on the LRU mechanism
>   * to evict old entries as the map fills up.
>   */
> -struct {
> -       __uint(type, BPF_MAP_TYPE_LRU_HASH);
> -       __type(key, __u32);
> -       __type(value, __u8);
> -       __uint(max_entries, 16384);
> -} flow_dscps SEC(".maps");
> +struct bpf_map_def SEC("maps") flow_dscps = {
> +       .type           = BPF_MAP_TYPE_LRU_HASH,
> +       .key_size       = sizeof(__u32),
> +       .value_size     = sizeof(__u8),
> +       .max_entries    = 16384,
> +};
>  
>  const volatile static int ip_only = 0;
> 

That change gives me the next error:
libbpf: map '' (legacy): static maps are not supported

Also speaks for itself...

> > Using the LLVM/Clang compiled object also doesn't work:
> > root at OpenWrt:/usr/lib/bpf# preserve-dscp wg0 eth0
> > libbpf: Error in bpf_create_map_xattr(flow_dscps):Operation not permitted(-1). Retrying without BTF.
> > libbpf: map 'flow_dscps': failed to create: Operation not permitted(-1)
> > libbpf: permission error while running as root; try raising 'ulimit -l'? current value: 512.0 KiB
> > libbpf: failed to load object 'preserve_dscp_kern.o'
> > Failed to load object
> >
> > Probably Kernel 5.4.124 is too old...?
> 
> Here I think the hint is in the error message ;)

Yep, I realized I had to increase it to ulimit to 2048...
With that at least the LLVM/Clang generated BPF object seems to load
properly, and I can load and unload it as expected.

> 
> >> An alternative to getting LLVM built as part of the OpenWrt toolchain is
> >> to just use the host clang to build the BPF binaries. It doesn't
> >> actually need to be cross-compiled with a special compiler, the BPF byte
> >> code format is the same on all architectures except for endianness, so
> >> just passing that to the host clang should theoretically be enough...
> >
> > I believe that having a way to build BPF objects compatible with the
> > target built-into our toolchain would be a huge step forward.
> > And given that gcc already get's pretty far, I think it'd be worth
> > fixing/patching what ever is missing (I haven't even tried GCC-11 yet)
> 
> For this example that might work (as noted above), but for other things
> BTF is a hard requirement, and I don't believe GCC supports that at all,
> sadly :(

It looks like this has changed very recently:

https://gcc.gnu.org/git/?p=gcc.git;a=commit;h=d5cf2b5db325fd5c053ca7bc8d6a54a06cd71124

> 
> > Find my staging tree including 'preserve-dscp' ready to play with:
> >
> > https://git.openwrt.org/?p=openwrt/staging/dangole.git;a=shortlog;h=refs/heads/gcc10-bpf
> >
> > Select 'Enable experimental features by default', but note that toolchain
> > doesn't build when selecting Linux 5.10 for x86, so you need to un-select
> > 'Use testing Kernel' if building for x86.
> > And have a look at the patch for allow building bpf-examples BPF objects
> > with GCC in package/network/utils/bpf-examples/patches
> >
> >
> >> 
> >> > In terms of kernel support: recent kernels don't build yet because of
> >> > gelf_getsymshndx, so we got to update libelf first for that. Recent
> >> > libelf doesn't seem to be an option yet on many of the build hosts we
> >> > currently support (Darwin and such).
> >> >
> >> > In terms of library support: our build of libbpf comes from Linux
> >> > release tarballs. There isn't yet a release supporting bpf_tc_attach,
> >> > the easiest would be to wait for Linux 5.13 to be released.
> >> 
> >> I used the libbpf TC loading support for convenience, but it's possible
> >> to load it using 'tc' as well without too much trouble (right now the
> >> userspace component sets a config variable before loading the program,
> >> but it can be restructured to not need that).
> >> 
> >> Alternatively, the bpf-examples repository is setup with a libbpf
> >> submodule that it can link statically against, so you could use that for
> >> now?
> >
> > I've updated to 5.13 + patches on top, so now it builds :)
> 
> Alright, that works.
> 
> > Library-embedding is a no-go for OpenWrt. Having different ABI-versions
> > of libraries installed simultanously works, so we can just ship with
> > a more recent version of libbpf.
> 
> Yeah, I wasn't suggesting it as a permanent solution, just so you could
> test it out :)

In the long run it would be great to have a somehow standardized and
reproducible way to build packages containing targetted BPF objects.
As having LLVM/Clang built-into OpenWrt has shown to be a huge amount
of work, I'm looking forward to GCC-12 which will support BTF from how
it looks by now...

> 
> >> > I (of course ;) also tried and spend almost a day looking for a
> >> > quick-and-dirty path for temporary deployment, so I could at least give
> >> > feedback -- bpf-examples also isn't exactly made to be cross-compiled
> >> > manually, so I have failed with that as well so far.
> >> 
> >> Heh, no, it isn't, really. Anything in particular you need to make this
> >> easier? We already added some bits to xdp-tools for supporting
> >> cross-compilation (and that shares some lineage with bpf-examples), so
> >> porting those over should not be too difficult.
> >
> > I found my way around, see the packaging for bpf-examples in the tree
> > (link above, at path stated above)
> 
> Right, I see. 

I have managed to test your solution and it seems to do the job.
Remaining issues:
 * What to do if there are many tunnels all sharing the same upstream
   interface? In this case I'm thinking of doing:
   preserve-dscp wg0 eth0
   preserve-dscp wg1 eth0
   preserve-dscp wg2 eth0
   ...
   But I'm unsure whether this is indented or if further details need
   to be implemented in order to make that work.

 * Once a wireguard interface goes down, one cannot unload the
   remaining program on the upstream interface, as
   preserve-dscp wg0 eth0 --unload
   would fail in case of 'wg0' having gone missing.
   What do you suggest to do in this case?

> 
> >> 
> >> See: https://github.com/xdp-project/xdp-tools/pull/78 and
> >> https://github.com/xdp-project/xdp-tools/issues/74
> >> 
> >> Unfortunately I don't have a lot of time to poke more at this right now,
> >> but feel free to open up an issue / pull request to the bpf-examples
> >> repository with any changes you need :)
> >
> > I guess I'll just go ahead then and package xdp-tools :)
> 
> That would be awesome! xdp-tools will definitely need BTF, though, so
> I'm afraid it'll need to be compiled with LLVM at this stage...

I'll probably move on doing other things for a while then and get back
to it once GCC-12 has been released...

Cheers

Daniel