How to enable containerd when using EKS managed node group

[bryantbiggs]

I know this may not be the best place to ask, but I wanted to give it a shot anyways.

Is there a way to simply just set environment variables when using EKS managed node groups (*WITHOUT specifying a custom AMI ID and therefore relying on the default user data bootstrap script provided by the EKS managed node group) and still enable contianerd?

You can see more details here, but it looks like in order to get containerd we have to go into some bash fu in the user data that we/users provide that gets pre-pended to the user data that the EKS managed node group service provides in order to get this to work:

#!/bin/bash
set -ex
cat <<-EOF > /etc/profile.d/bootstrap.sh
export CONTAINER_RUNTIME="containerd"
export USE_MAX_PODS=false
export KUBELET_EXTRA_ARGS="--max-pods=110"
EOF
# Source extra environment variables in bootstrap script
sed -i '/^set -o errexit/a\\nsource /etc/profile.d/bootstrap.sh' /etc/eks/bootstrap.sh

I just wanted to see if it were possible to do the following instead do (at least, thats what the "API" looks like when viewing the bootstrap.sh script:

export CONTAINER_RUNTIME="containerd"
export USE_MAX_PODS=false
export KUBELET_EXTRA_ARGS="--max-pods=110"

thank you!

[cartermckinnon]

For this specific case, yes; this is the best way to accomplish it. The upcoming 1.22 release will make such workarounds unnecessary, as containerd will become the default runtime.

[cartermckinnon]

Sorry, my info was outdated -- the change to containerd has been pushed to the 1.23 release; there is more context available in the docs.

[stevehipwell]

@bryantbiggs @cartermckinnon I'm planning on opening a PR in this repo to update bootstrap.sh with a known file to source when present. This will be a no-op unless someone creates the file to set these variables (or other variables such as dynamic labels).

[lobeck]

This does not seem to work anymore.

It only uses the args passed through the call to bootstrap.sh and the environment variable gets overwritten.

Edit: To specify: My userdata only contains the preparatory work, the bootstrap.sh call is added by EKS internally. But EKS adds --kubelet-extra-args as flag and thus the environment variable is just overwritten and has no effect.

Does anyone have a simple way to work around this or do I need to patch bootstrap.sh to get it working?

[stevehipwell]

@lobeck take a look at #855. You need to modify bootstap.sh until this capability is added, either to source another persisted file or to directly set the variable.

[lobeck]

@stevehipwell thank you. got it working with

sed -i 's/KUBELET_EXTRA_ARGS=$2/KUBELET_EXTRA_ARGS="$2 $KUBELET_EXTRA_ARGS"/' /etc/eks/bootstrap.sh

It's not nice, but get's the job done

[stevehipwell]

That's what #855 is meant to make a lot cleaner by allowing the passing of env variables between cloud init steps.

[lobeck]

I know and that patch is already in place.

cat <<-EOF > /etc/profile.d/bootstrap.sh
export CONTAINER_RUNTIME="containerd"
export KUBELET_EXTRA_ARGS="--runtime-request-timeout 600s"
EOF
# Source extra environment variables in bootstrap script
sed -i '/^set -o errexit/a\\nsource /etc/profile.d/bootstrap.sh' /etc/eks/bootstrap.sh
sed -i 's/KUBELET_EXTRA_ARGS=$2/KUBELET_EXTRA_ARGS="$2 $KUBELET_EXTRA_ARGS"/' /etc/eks/bootstrap.sh

Unfortunately with terraform and EKS itself, it needs to merge both environment variables and command line arguments.

I added stuff through the environment variables and EKS injects all the labels as command line flags on bootstrap.sh

My sed only patches the line in the argument parser, so that in case both are specified, they get merged. If there are no flags set, the environment variable will be used.

[stevehipwell]

@lobeck if you look at terraform-aws-modules/terraform-aws-eks#1789, specifically the new linux_mng_merge_user_data.tpl file you'll see my updated version of how this should work; effectively you just need to set env variables to fully configure bootstrap. The code in the initial issue comes from the implementation I added to an earlier version of the terraform-aws-eks module which was designed to work with other implementations.

[lobeck]

Awesome, can't wait for this to get merged, so I can remove my crufty patches :D

[rshad]

This should be tricked by adding the following line to the user data:

sed -i 's/CONTAINER_RUNTIME:-dockerd/CONTAINER_RUNTIME:-containerd/' /etc/eks/bootstrap.sh

[Ga13Ou]

#!/bin/bash
set -ex
cat <<-EOF > /etc/profile.d/bootstrap.sh
export CONTAINER_RUNTIME="containerd"
export USE_MAX_PODS=false
export KUBELET_EXTRA_ARGS="--max-pods=110"
EOF
# Source extra environment variables in bootstrap script
sed -i '/^set -o errexit/a\\nsource /etc/profile.d/bootstrap.sh' /etc/eks/bootstrap.sh

Did anyone had the Disk capacity warning problem when tried to migrate to Containerd. Every solution I use I get this error

Warning InvalidDiskCapacity 9m25s kubelet invalid capacity 0 on image filesystem