Failure to load images with kind v0.26.0 and kindes/node:v1.32.1

[sorindumitru]

What happened:

Our CI tries to figure out the latest versions of kind and kindest/node to run integration tests against various k8s versions. After the image with v1.32.1 (roughly, I know it worked on Tuesday the 21st, but not on Thursday the 23rd) we started seeing CI failures for the combination of kind 0.26.0 and kindes/node v1.32.1. The error occurs when trying to load an image into the cluster:

[2025-01-24T08:51:40Z] executing 00-setup...
Ensuring kind version v0.27.0-alpha is available...
Want kubectl version: v1.32.1
Have kubectl version: v1.32.1
[2025-01-24T08:51:40Z] starting cluster...
Creating cluster "k8stest" ...
 ✓ Ensuring node image (kindest/node:v1.32.1) 🖼
 ✓ Preparing nodes 📦
 ✓ Writing configuration 📜
 ✓ Starting control-plane 🕹️
 ✓ Installing CNI 🔌
 ✓ Installing StorageClass 💾
Set kubectl context to "kind-k8stest"
You can now use your cluster with:

kubectl cluster-info --context kind-k8stest

Have a nice day! 👋
[2025-01-24T08:51:55Z] loading container images...
Image: "spire-server:latest-local" with ID "sha256:83de609b1ad3b445d26a9355b2926140f1bb0010649aa9858a270bd2acb14870" not yet present on node "k8stest-control-plane", loading...
ERROR: failed to detect containerd snapshotter
Stack Trace:
sigs.k8s.io/kind/pkg/errors.New
        /home/sorin/github/kubernetes-sig/kind/pkg/errors/errors.go:28
sigs.k8s.io/kind/pkg/cluster/nodeutils.parseSnapshotter
        /home/sorin/github/kubernetes-sig/kind/pkg/cluster/nodeutils/util.go:107
sigs.k8s.io/kind/pkg/cluster/nodeutils.getSnapshotter
        /home/sorin/github/kubernetes-sig/kind/pkg/cluster/nodeutils/util.go:97
sigs.k8s.io/kind/pkg/cluster/nodeutils.LoadImageArchive
        /home/sorin/github/kubernetes-sig/kind/pkg/cluster/nodeutils/util.go:81
sigs.k8s.io/kind/pkg/cmd/kind/load/docker-image.loadImage
        /home/sorin/github/kubernetes-sig/kind/pkg/cmd/kind/load/docker-image/docker-image.go:205
sigs.k8s.io/kind/pkg/cmd/kind/load/docker-image.runE.func1
        /home/sorin/github/kubernetes-sig/kind/pkg/cmd/kind/load/docker-image/docker-image.go:190
sigs.k8s.io/kind/pkg/errors.UntilErrorConcurrent.func1
        /home/sorin/github/kubernetes-sig/kind/pkg/errors/concurrent.go:30
runtime.goexit
        /usr/lib/go/src/runtime/asm_amd64.s:1700
[2025-01-24T08:51:56Z] step 00-setup failed

See also our issue for the build failing spiffe/spire#5812

Doing some more experiments from main I can see that:

• main works
• 586b038 works
• 82a216b fails
• Also seems to work with kindest/node:v1.31.4 and kind v0.26.0

So it's safe to say that 586b038 is what makes it work, but what makes it break is the new image for v1.32.1

What you expected to happen:

To be able to load images.

How to reproduce it (as minimally and precisely as possible):

kind create cluster --name loadimage --image kindest/node:v1.32.1
kind load docker-image kindest/node:v1.32.1 --name loadimage

Anything else we need to know?:

Environment:
Linux

• kind version: (use kind version): v0.26.0
• Runtime info: (use docker info, podman info or nerdctl info):

Client:
 Version:    27.3.1
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  0.19.3
    Path:     /usr/lib/docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  2.32.4
    Path:     /usr/lib/docker/cli-plugins/docker-compose

Server:
 Containers: 10
  Running: 4
  Paused: 0
  Stopped: 6
 Images: 39
 Server Version: 27.3.1
 Storage Driver: overlay2
  Backing Filesystem: btrfs
  Supports d_type: true
  Using metacopy: true
  Native Overlay Diff: false
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: systemd
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: c507a0257ea6462fbd6f5ba4f5c74facb04021f4.m
 runc version:
 init version: de40ad0
 Security Options:
  seccomp
   Profile: builtin
  cgroupns
 Kernel Version: 6.12.9-arch1-1
 Operating System: Arch Linux
 OSType: linux
 Architecture: x86_64
 CPUs: 16
 Total Memory: 30.66GiB
 Name: bitsy
 ID: 96781e78-8781-42e7-9c94-52924e7b3db0
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

• OS (e.g. from /etc/os-release):
  Arch Linux (my laptop)/Ubuntu 22.04 for CI
• Kubernetes version: (use kubectl version):
  v1.32.1
• Any proxies or other special environment settings?:
  No

[aojea]

/assign @BenTheElder

since he was discussing this with @AkihiroSuda recently during the upgrade to containerd 2.0, that seems that fixed the problem

[AkihiroSuda]

Looks like the kindest/node image was updated to containerd v2 ahead of the new release of the kind cmd.

The kind cmd should have a new release, or, the image should be reverted

[sorindumitru]

The kind cmd should have a new release, or, the image should be reverted

Thanks, that was my thinking as well. I think either of them would work for us.

[BenTheElder]

FYI: new images like this are not necessarily supported with old releases (see the release notes and the docs which both clearly warn about image selection)

This one should be working though.

[BenTheElder]

The kind cmd should have a new release, or, the image should be reverted

We will do a new release. But it is not a safe assumption that you can use any image with any release and every release's notes discuss this with the listed images as does the quick start docs section about changing versions / images.

Further those docs warn that you should use images by digest for security and to avoid this.

/close
/remove-kind bug

[k8s-ci-robot]

@BenTheElder: Closing this issue.

In response to this:

The kind cmd should have a new release, or, the image should be reverted

We will do a new release. But it is not a safe assumption that you can use any image with any release and every release's notes discuss this with the listed images as does the quick start docs section about changing versions / images.

/close
/remove-kind bug

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[BenTheElder]

Also noting that this does not collide with any previously existing tags (v1.32.0 was the tag available for 0.27? but it could have given we advertise the digests and instruct users to use the digests for exact images

[BenTheElder]

Our CI tries to figure out the latest versions of kind and kindest/node to run integration tests against various k8s versions.

You should scrape the release notes instead of the docker hub tags. We can and have made breaking changes before by necessity to fix bugs. We avoid them where possible but don't guarantee it. This is one such example, we need to keep up with containerd.

Really you should just select specific images and not scrape anything. These images run with full privileges and pulling random unvetted images is a serious risk, if you pull by digest then even if we are temporarily compromised you cannot be impacted.

[BenTheElder]

There is some possible discussion in open issues of using another tag scheme in the future, but the discussion has yet to settle.

[kaovilai]

You should scrape the release notes

Is release notes in a reliable format that you can regex in a script?

[kaovilai]

We are currently doing something like this to get latest k8s versions to use in CI. (here)

❯ wget -q -O - "https://hub.docker.com/v2/namespaces/kindest/repositories/node/tags?page_size=50" | grep -o '"name": *"[^"]*' | grep -o '[^"]*$' | grep -v -E "alpha|beta|1\.3[2-9]\.[1-9]" | grep -E "v[1-9]\.(2[5-9]|[3-9][0-9])" | awk -F. '{if(!a[$1"."$2]++)print $1"."$2"."$NF}' | sort -r | sed s/v//g | jq -R -c -s 'split("\n")[:-1]'
["1.32.0","1.31.4","1.30.8","1.29.12","1.28.15","1.27.16","1.26.15","1.25.16"]

[BenTheElder]

We're still hoping for another containerd release with some some fixes for testing Kubernetes, otherwise I'm personally out a lot at the moment for personal reasons so it may be a bit before we officially release with this image.

(To be clear, I am not the only person that can release, but I can't speak on anyone else's behalf)

[BenTheElder]

https://github.com/kubernetes-sigs/kind/releases/tag/v0.27.0 is out

[kaovilai]

woot.

perhaps I can use default node image of cli as upper bound in my prior script.