[RFC] Linux Network Devices #4538

aojea · 2024-11-21T22:02:46Z

Draft implementation of opencontainers/runtime-spec#1271

It implements the new proposal to the OCI spec to be able to specify Network Devices that get attached/detached from the containers

tests/integration/netdev.bats

kad

LGTM. We are also interested in this use case for our accelerator devices.

libcontainer/network_linux.go

tests/integration/netdev.bats

libcontainer/network_linux.go

libcontainer/configs/validate/validator.go

libcontainer/network_linux.go

libcontainer/factory_linux.go

rata · 2025-02-25T13:05:57Z

@aojea friendly reminder that this should be ready soon, we are cutting 1.3.0-rc.1 soon (maybe this week). The spec part is still not sure when it will be merged?

aojea · 2025-02-28T23:25:14Z

@aojea friendly reminder that this should be ready soon, we are cutting 1.3.0-rc.1 soon (maybe this week). The spec part is still not sure when it will be merged?

@rata I'm not familiar with the processes in this ecosystem, should I do something? I'm waiting for the spec to merge

rata · 2025-03-03T10:51:58Z

@aojea I don't think it's much different from k8s or most projects. I think you need to drive alignment to merge the runtime-spec PR. But we can't merge this in runc if it depends on a spec change.

aojea · 2025-03-03T11:02:00Z

@aojea I don't think it's much different from k8s or most projects. I think you need to drive alignment to merge the runtime-spec PR. But we can't merge this in runc if it depends on a spec change.

that is why I thought, thanks for confirming

tests/integration/netdev.bats

rata

@aojea I think the bug we were talking on slack is a bug in the validator, see the comments for more info.

rata · 2025-03-03T13:43:15Z

libcontainer/configs/validate/validator.go

+	if config.Namespaces.IsPrivate(configs.NEWNET) {
+		return errors.New("unable to move network devices without a NET namespace")
+	}


This seems to be wrong. This checks if it has a Path set. But this should work for any network namespace.

When you remove this, the rest of the code might fail if you need the netns fd. In that case, I think you should get the ns fs from /proc/<pid>/ns/net (you open that path and the fd you get is a netns fd), or even better if you can use the pidfd as that doesn't have races and those pains.

IMHO, we should implement both: use pidfd whenever possible and fallback to /proc/<pid>/ns/net when that is not available, and handle the races properly.

tests/integration/netdev.bats

rata · 2025-03-03T13:55:49Z

tests/integration/netdev.bats

It will be great if you can add a test that also uses a user namespace. See some examples here: tests/integration/userns.bats

I'm not familiar with user namespaces but the topic was also brought up during the review and it seems was decided to not included them, is not the network different on those environments? https://github.com/rootless-containers/rootlesskit/blob/master/docs/network.md

@aojea where was it brought up, do you have links? I'd like to see why it wasn't included.

The link you mention is about rootless containers, that is something different. That is to NOT run the runtime as root on the host. Rootless containers uses userns too, that is why it is confusing.

I meant the container created has a user namespace, the same way it has a network, pid, etc. namespace. It doesn't change at all how the runtime runs, it is a namespace use in the container created. The container runtime runs as root, etc. It's a completely different thing.

ahhhh ok, then I will add those tests too

aojea · 2025-03-03T22:10:27Z

libcontainer/state_linux.go

+	if nsPath == "" {
+		if c.initProcess.pid() > 0 {
+			nsPath = fmt.Sprintf("/proc/%d/ns/net", c.initProcess.pid())
+		} else {
+			return nil
+		}


@rata I need help here, I can not find the best place to restore the network devices when the namespace is created by runc, it seems it has to be at one point where the init process is still running

@aojea what do you mean "the init process is still running"? Which process?

state_linux.go is probably to know the state of running containers, you should check to standard_init_linux.go, for example

aojea · 2025-03-04T10:10:56Z

libcontainer/state_linux.go

@@ -48,6 +49,14 @@ func destroy(c *Container) error {
 		// Likely to fail when c.config.RootlessCgroups is true
 		_ = signalAllProcesses(c.cgroupManager, unix.SIGKILL)
 	}
+
+	// Likely to fail if the init process no longer exist.


I only found this place to hook for restoring these devices, but since the start process may not exist , when the network namespace is not provided it is not able to find the network namespace ... please advice

See this: #4538 (comment)

state_linux.go is not the right place, for sure. What I say in the comment seems to have a better chance. But I don't know the details of moving the network interfaces to know exactly where is the right place to hook it up, sorry :(

any point where we can get the network namespace and the initial process has not started yet

What do you refer to with "the initial process" ? The container process? Or the runc init processes?

All runc code runs before the container process, once we exec the container runc is out of the picture.

Assuming you mean the container process, the namespaces are created in nsexec.c, there are a lot of things done after that in go too. Like the mountToRootfs(), etc.

I have not look in detail to suggest where to do it, but take a look and just add it somewhere. People will review and it can be moved in the worst case :)

Now that I think about it, the spec has two subcommands that are relevant here: create and start.

creates creates all the ns, etc. and start starts the container process. I don't know when CNI is being called by upper container runtimes (like containerd or crio), but I guess we should move the interface at that point (whatever it is, create or start).

I don't know when CNI is being called by upper container runtimes (like containerd or crio), but I guess we should move the interface at that point (whatever it is, create or start)

in containerd the network namespace is created and the CNI is called to provide IPs and the network interface, once that is set the containers starts to be created

the setup seems to be in the correct place now, before the Prestart hook

Change-Id: Id54c22cb5652c77b4a3c3bd4e46e96388be1cc11 Signed-off-by: Antonio Ojea <aojea@google.com>

Implement support for passing Linux Network Devices to the container network namespace. The network device is passed during the creation of the container, before the process is started. It implements the logic defined in the OCI runtime specification. Change-Id: I190d5c444f03e65bbbfe5b4bc90809c0ad0a2017 Signed-off-by: Antonio Ojea <aojea@google.com>

aojea mentioned this pull request Nov 21, 2024

Define Linux Network Devices opencontainers/runtime-spec#1271

Open

aojea force-pushed the netdevices branch 2 times, most recently from bdb31c1 to 0b771ca Compare December 2, 2024 15:28

aojea commented Dec 2, 2024

View reviewed changes

tests/integration/netdev.bats Outdated Show resolved Hide resolved

aojea force-pushed the netdevices branch 2 times, most recently from 07d3b0b to 3833056 Compare December 2, 2024 15:40

kad reviewed Dec 10, 2024

View reviewed changes

lifubang reviewed Dec 10, 2024

View reviewed changes

libcontainer/network_linux.go Outdated Show resolved Hide resolved

tests/integration/netdev.bats Outdated Show resolved Hide resolved

aojea force-pushed the netdevices branch 2 times, most recently from 67f12e0 to d114afe Compare December 12, 2024 07:15

aojea commented Dec 17, 2024

View reviewed changes

libcontainer/network_linux.go Outdated Show resolved Hide resolved

aojea force-pushed the netdevices branch 2 times, most recently from ec90a02 to f4f5d02 Compare December 20, 2024 12:11

aojea force-pushed the netdevices branch 2 times, most recently from 735f9d5 to ce1f612 Compare January 14, 2025 09:47

aojea force-pushed the netdevices branch 4 times, most recently from 6262c5e to c530772 Compare February 6, 2025 09:55

kolyshkin reviewed Feb 7, 2025

View reviewed changes

libcontainer/configs/validate/validator.go Outdated Show resolved Hide resolved

kolyshkin reviewed Feb 7, 2025

View reviewed changes

libcontainer/configs/validate/validator.go Outdated Show resolved Hide resolved

kolyshkin reviewed Feb 7, 2025

View reviewed changes

libcontainer/configs/validate/validator.go Outdated Show resolved Hide resolved

kolyshkin reviewed Feb 7, 2025

View reviewed changes

libcontainer/network_linux.go Outdated Show resolved Hide resolved

kolyshkin reviewed Feb 8, 2025

View reviewed changes

libcontainer/factory_linux.go Outdated Show resolved Hide resolved

aojea force-pushed the netdevices branch 5 times, most recently from 4380e86 to f53d263 Compare February 10, 2025 21:27

aojea force-pushed the netdevices branch from f53d263 to a8a2045 Compare March 3, 2025 13:18

rata reviewed Mar 3, 2025

View reviewed changes

tests/integration/netdev.bats Show resolved Hide resolved

rata reviewed Mar 3, 2025

View reviewed changes

aojea force-pushed the netdevices branch 2 times, most recently from f74c032 to 17c7056 Compare March 3, 2025 22:09

aojea commented Mar 3, 2025

View reviewed changes

aojea force-pushed the netdevices branch 2 times, most recently from 9f4b286 to 20f65c5 Compare March 3, 2025 22:37

aojea commented Mar 4, 2025

View reviewed changes

aojea added 2 commits March 7, 2025 14:44

TEMP vendor linx netdevices OCI spec

78de932

Change-Id: Id54c22cb5652c77b4a3c3bd4e46e96388be1cc11 Signed-off-by: Antonio Ojea <aojea@google.com>

aojea force-pushed the netdevices branch from 20f65c5 to 0655187 Compare March 7, 2025 13:44

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[RFC] Linux Network Devices #4538

[RFC] Linux Network Devices #4538

aojea commented Nov 21, 2024 •

edited

Loading

kad left a comment

rata commented Feb 25, 2025

aojea commented Feb 28, 2025

rata commented Mar 3, 2025

aojea commented Mar 3, 2025

rata left a comment

rata Mar 3, 2025 •

edited

Loading

rata Mar 3, 2025

aojea Mar 3, 2025

rata Mar 3, 2025 •

edited

Loading

aojea Mar 3, 2025

aojea Mar 3, 2025

rata Mar 4, 2025

aojea Mar 4, 2025

rata Mar 4, 2025

aojea Mar 4, 2025

rata Mar 4, 2025

rata Mar 4, 2025

rata Mar 4, 2025

aojea Mar 5, 2025 •

edited

Loading

aojea Mar 5, 2025

[RFC] Linux Network Devices #4538

Are you sure you want to change the base?

[RFC] Linux Network Devices #4538

Conversation

aojea commented Nov 21, 2024 • edited Loading

kad left a comment

Choose a reason for hiding this comment

rata commented Feb 25, 2025

aojea commented Feb 28, 2025

rata commented Mar 3, 2025

aojea commented Mar 3, 2025

rata left a comment

Choose a reason for hiding this comment

rata Mar 3, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rata Mar 3, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

aojea Mar 5, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

aojea commented Nov 21, 2024 •

edited

Loading

rata Mar 3, 2025 •

edited

Loading

rata Mar 3, 2025 •

edited

Loading

aojea Mar 5, 2025 •

edited

Loading