Custom labels for native and go

[gnurizen]

Add custom labels for native processes utilizing the PolarSignals
TLS based native custom labels library:

https://github.com/polarsignals/custom-labels

Go custom labels are auto extracted from the go env pprof labels set.

Go 1.23 and 1.24 style map and slice approaches are both supported.

Label keys are 16 bytes and values are 48 bytes. Each sample can have
up to 10 labels which are stored in the Trace object adding 640 bytes.

Unit test added for go custom labels and an Rust based example for
the native custom labels is added.

Co-authored-by: umanwizard brennan@umanwizard.com

[gnurizen]

I need to sort out the arm failure but this is ready for an initial review, thanks in advance!

[gnurizen]

Okay seems to be passing all tests and has test coverage for native and golang labels.

[jordanisaacs]

This is awesome, thanks for working on it! (Closes #388). One concern for native libraries is the custom labels API uses a single thread local stack that does allocations.

The nice thing about the strobemeta API is it lets the user set up multiple TLS of different types (map/string/int). The program however has to be profiler aware and register with profiler using a ebpf map for configuration. The upside is non-allocating tag values. Data storage is just the thread local.

My takeaway is the custom labels library can be thought of as auto-discovery of a map type.

Can the two approaches be combined, auto-discovery and multiple TLS values? Would need to auto-discover the the multiple TLS locations/types.

edit: Read a bit more, into how the profiler discovers the TLS value. Allowing a config file that lets you specify TLS names+types would work. The custom label API would also have to change to let you declare your own thread locals. The tag name of non map types could be the name of the TLS.

[gnurizen]

Thanks for the feedback @jordanisaacs! Curious why doing allocations is a deal breaker, our thought was that labels would be setup at the beginning of fairly long running tasks and the allocations would be minimal.

Having multiple TLS variables of different types shouldn't be too hard to add to this implementation, support for single TLS strings and ints that allowed for zero-allocation library free usage would be nice. Where does this TLS names+types config live? Maybe in the binary as a special debug elf section?

[jordanisaacs]

Not a deal breaker but something I think would be really useful. One use case I can imagine is having an enum that represents the phases of a query execution. Updating the query phase is simply assigning an int instead of allocating a string.

I like the ELF idea a lot. The ELF section should be non-debug. So stripped binaries still support custom metadata. Could it just be a list of addresses of the TLS? The initial keys would just be the TLS address. Since symbolization is already done elsewhere for native libraries, symbolization would also turn addresses into their names.

[fabled]

Would it be possible to split this to separate commits or even PRs?
E.g

• changes to libraries
• support for native
• support for golang

Also, the golang interpreter needs to be coordinated. Also #408 is adding the same interpreter for different purposes with different code.

[gnurizen]

Yeah I can break it up, I feel like the go stuff should be pretty uncontroversial so maybe I should structure it as a go labels PR with a follow on native labels PR? Although with all the shared code the first one will still be pretty big. Its unclear to me what "changes to libraries" means.

[gnurizen]

Here's another PR with just the go custom labels:
#409

[gnurizen]

I'll close this one and open another one with the native custom labels when/if the go labels one lands.