Add automemlimit

[alexbozhenko]

Summary

The problem:

A Guide to the Go Garbage Collector suggests:

Do take advantage of the memory limit when the execution environment of your Go program is entirely within your control, and the Go program is the only program with access to some set of resources (i.e. some kind of memory reservation, like a container memory limit).
A good example is the deployment of a web service into containers with a fixed amount of available memory.
In this case, a good rule of thumb is to leave an additional 5-10% of headroom to account for memory sources the Go runtime is unaware of.

When running in Kubernetes, the memory limit set using cgroups

Let's use the library that can automatically set the GOMEMLIMIT to a set percentage of memory limit of the cgroup

Reproducibe test plan:

1. When no GOMEMLIMIT is set, the effective value is the default, math.MaxInt64

# go build .

# ./nats-server 
...
[81117] 2025/02/20 12:59:21.150405 [INF] Server is ready
2025/02/20 12:59:21 INFO memory is not limited, skipping package=github.com/KimMachineGun/automemlimit/memlimit
[81117] 2025/02/20 12:59:21.151533 [INF] Effective GOMEMLIMIT: 9223372036854775807 bytes

2. When env var is set, its value is respected:

# GOMEMLIMIT=1000MiB ./nats-server 
[81345] 2025/02/20 13:00:50.034879 [INF] Server is ready
2025/02/20 13:00:50 INFO GOMEMLIMIT is already set, skipping package=github.com/KimMachineGun/automemlimit/memlimit GOMEMLIMIT=1000MiB
[81345] 2025/02/20 13:00:50.035615 [INF] Effective GOMEMLIMIT: 1048576000 bytes

Build a docker image for test:

goreleaser release --snapshot --clean -f .goreleaser-nightly.yml 

3. When running in docker, by default no limit is set:

# docker run synadia/nats-server:add_automemlimit
...
[1] 2025/02/20 21:03:10.188627 [INF] Server is ready
[1] 2025/02/20 21:03:10.188685 [INF] Cluster name is 9QPBj9CtvlZlE9b8iyUn3a
[1] 2025/02/20 21:03:10.188829 [WRN] Cluster name was dynamically generated, consider setting one
[1] 2025/02/20 21:03:10.188923 [INF] Listening for route connections on 0.0.0.0:6222
2025/02/20 21:03:10 INFO memory is not limited, skipping package=github.com/KimMachineGun/automemlimit/memlimit
[1] 2025/02/20 21:03:10.189336 [INF] Effective GOMEMLIMIT: 9223372036854775807 bytes

4. When running in a container with a memory limit:

# docker run --cpus="4.0" --memory="2000m"  --name nats_container synadia/nats-server:add_automemlimit
[1] 2025/02/20 21:03:58.776768 [INF] Server is ready
[1] 2025/02/20 21:03:58.776869 [INF] Cluster name is ffq6b2J0D4cUuImY6PcyRA
[1] 2025/02/20 21:03:58.776899 [WRN] Cluster name was dynamically generated, consider setting one
[1] 2025/02/20 21:03:58.776986 [INF] Listening for route connections on 0.0.0.0:6222
2025/02/20 21:03:58 INFO GOMEMLIMIT is updated package=github.com/KimMachineGun/automemlimit/memlimit GOMEMLIMIT=1677721600 previous=9223372036854775807
[1] 2025/02/20 21:03:58.777953 [INF] Effective GOMEMLIMIT: 1677721600 bytes

The GOMEMLIMIT is set to 80% of the cgroup memory limit, as expected.

# cat  /sys/fs/cgroup/system.slice/docker-$(docker inspect --format="{{.Id}}" nats_container).scope/memory.max
2097152000

Signed-off-by: Alex Bozhenko alex@synadia.com

[alexbozhenko]

Probably do not want to use slog here, but I was not sure how to plug it to the server logger, since memlimit.WithLogger expects slog logger.

[wallyqs]

I think we would need to make the server logger implement the Handler interface from slog, then here call slog.New(s.Logger())

[wallyqs]

this dependency is not tagged nor released in 4-7 years... pinged the author on whether they could tag one

[alexbozhenko]

The ratio should be a config option?

[wallyqs]

yes I think sudden default to 0.8 might be too sudden as a knob, in the config would be better. With default GOMEMLIMIT you cannot do fractions but would be interesting if this dependency supports doing 0.85 for example.