Struggling to understand how to configure containerd.toml for a private registry #1482

DazWilkin · 2020-05-15T20:51:06Z

I found this and #835 but I receive unauthorized errors when trying to pull an image from GCR:

level=error msg="PullImage "gcr.io/[redacted]/hello-world:latest" failed" error="failed to resolve image "gcr.io/[redacted]/hello-world:latest": no available registry endpoint: unexpected status code https://gcr.io/v2/[redacted]/hello-world/manifests/latest: 401 Unauthorized"

I'm using MicroK8s with Krustlet and the Krustlet does not yet support imagePullSecrets so I'd like to configure MicroK8s' containerd to authenticate.

Per the docs, I'm replicating the approach used with Docker:

gcloud auth print-access-token \
| docker login \
-u oauth2accesstoken \
--password-stdin \
https://gcr.io

more ${HOME}/.docker/config.json
{
	"auths": {
		"https://gcr.io": {
			"auth": "b2F1dGgy..."
		}
	}
}

I appreciate that I'll have limited time to use the token but...

I tried:

...
[plugins]
  ...
  [plugins.cri]
    ...
    [plugins.cri.registry]
      [plugins.cri.registry.mirrors]
        [plugins.cri.registry.mirrors."docker.io"]
          endpoint = ["https://registry-1.docker.io"]
        [plugins.cri.registry.mirrors."local.insecure-registry.io"]
          endpoint = ["http://localhost:32000"]
        [plugins.cri.registry.mirrors."gcr.io"]
          endpoint = ["https://gcr.io"]
      [plugins.cri.registry.configs]
        [plugins.cri.registry.configs.auths]
          [plugins.cri.registry.configs.auths."https://gcr.io"]
            auth = "b2F1dGgy..."
...

I'm restarting MicroK8s after changes.

Using Docker against the registry, works:

docker image ls gcr.io/[redacted]/hello-world
REPOSITORY          TAG                 IMAGE ID            CREATED

docker pull gcr.io/[redacted]/hello-world
Using default tag: latest
latest: Pulling from [redacted]/hello-world
Digest: sha256:90659bf80b44ce6be8234e6ff90a1ac34acbeb826903b02cfa0da11c82cbc042
Status: Downloaded newer image for gcr.io/[redacted]/hello-world:latest

docker image ls gcr.io/[redacted]/hello-world
REPOSITORY                      TAG                 IMAGE ID            CREATED
gcr.io/[redacted]/hello-world   latest              bf756fb1ae65        4 months ago

Any help would be appreciated.

The text was updated successfully, but these errors were encountered:

DazWilkin · 2020-05-15T21:09:25Z

Thank you.

The node's kubelet (Krustlet) does not (yet) implement using imagePullSecrets.

Am I unable to do this by some form of the configuration I show?

mikebrow · 2020-05-15T21:12:03Z

hmm :)

mikebrow · 2020-05-15T21:13:19Z

let's see the whole config

DazWilkin · 2020-05-15T21:17:49Z

It's mostly the default configuration created by MicroK8s:

oom_score = 0

[grpc]
  uid = 0
  gid = 0
  max_recv_message_size = 16777216
  max_send_message_size = 16777216

[debug]
  address = ""
  uid = 0
  gid = 0

[metrics]
  address = "127.0.0.1:1338"
  grpc_histogram = false

[cgroup]
  path = ""

[plugins]
  [plugins.cgroups]
    no_prometheus = false
  [plugins.cri]
    stream_server_address = "127.0.0.1"
    stream_server_port = "0"
    enable_selinux = false
    sandbox_image = "k8s.gcr.io/pause:3.1"
    stats_collect_period = 10
    systemd_cgroup = false
    enable_tls_streaming = false
    max_container_log_line_size = 16384
    [plugins.cri.containerd]
      snapshotter = "overlayfs"
      no_pivot = false
      [plugins.cri.containerd.default_runtime]
        runtime_type = "io.containerd.runtime.v1.linux"
        runtime_engine = ""
        runtime_root = ""
      [plugins.cri.containerd.untrusted_workload_runtime]
        runtime_type = ""
        runtime_engine = ""
        runtime_root = ""
    [plugins.cri.cni]
      bin_dir = "${SNAP}/opt/cni/bin"
      conf_dir = "${SNAP_DATA}/args/cni-network"
      conf_template = ""
    [plugins.cri.registry]
      [plugins.cri.registry.mirrors]
        [plugins.cri.registry.mirrors."docker.io"]
          endpoint = ["https://registry-1.docker.io"]
        [plugins.cri.registry.mirrors."local.insecure-registry.io"]
          endpoint = ["http://localhost:32000"]
        [plugins.cri.registry.mirrors."gcr.io"]
          endpoint = ["https://gcr.io"]
      [plugins.cri.registry.configs]
        [plugins.cri.registry.configs.auths]
          [plugins.cri.registry.configs.auths."https://gcr.io"]
            auth = "b2F1dGgy..."
  [plugins.diff-service]
    default = ["walking"]
  [plugins.linux]
    shim = "containerd-shim"
    runtime = "${RUNTIME}"
    runtime_root = ""
    no_shim = false
    shim_debug = true
  [plugins.scheduler]
    pause_threshold = 0.02
    deletion_threshold = 0
    mutation_threshold = 100
    schedule_delay = "0s"
    startup_delay = "100ms"

NOTE I've redacted the token value.

NOTE I was unsure whether the hierarchy needed to be explicitly defined e.g. [plugins.cri.registry.configs] and [plugins.cri.registry.configs.auth]

In the comment reply that I referenced, a different value was used [plugins.cri.registry.auths] (no configs)

mikebrow · 2020-05-15T21:20:11Z

[plugins.cri.registry.auths]
[plugins.cri.registry.auths."https://gcr.io"]
username = ""
password = ""
auth = ""
identitytoken = ""

mikebrow · 2020-05-15T21:21:05Z

	 	// Username is the username to login the registry.
Username string `toml:"username" json:"username"`
// Password is the password to login the registry.
Password string `toml:"password" json:"password"`
// Auth is a base64 encoded string from the concatenation of the username,
// a colon, and the password.
Auth string `toml:"auth" json:"auth"`
// IdentityToken is used to authenticate the user and get
// an access token for the registry.
IdentityToken string `toml:"identitytoken" json:"identitytoken"

mikebrow · 2020-05-15T21:23:29Z

seems to be missing v2

DazWilkin · 2020-05-15T21:24:23Z

Hmmm... this suggests that the auth value from e.g. ${HOME}/.docker/config.json. is not the same as this auth value. I would, e.g.:

gcloud auth print-access-token \
| docker login -u oauth2accesstoken --password-stdin https://gcr.io

This updates and entry in ${HOME}/.docker/config.json e.g.

{
	"auths": {
		"https://gcr.io": {
			"auth": "b2F1dGgy..."
		}
	}
}

And I've been mirroring the same value in the containerd TOML.

Yes, I am confused by the spec but it suggests that I don't need to explicit state the v2!?

I will look into it some more.

Thank you!

DazWilkin · 2020-05-15T21:28:03Z

I'm going to take my dog for a swim...

Will look at this tomorrow.

Thanks for looking into it for me!

DazWilkin · 2020-05-16T00:37:08Z

I've tried most permutations of username|password|auth and I'm unsuccessful :-(

mikebrow · 2020-05-16T15:44:38Z

Try this:
canonical/microk8s#990 (comment)

mikebrow · 2020-05-16T15:47:15Z

if that works out for you I'll fix the docs ..

DazWilkin · 2020-05-16T17:39:55Z

It does not :-(

But, I mostly understand why that approach should work. It seems redundant to have to provide the username and password twice; once directly and then a second time base64 encoded.

I'd prefer to not use a long-lived token (from a service account) as is done in that example (see Authentication Methods).

The configuration is confusing. In this (and others) examples, the registry auth is referenced as:

registry.auths

But, in the sources:

cri/pkg/config/config.go

Line 181 in 8252e54

Registry Registry `toml:"registry" json:"registry"`

cri/pkg/config/config.go

Line 144 in 8252e54

Configs map[string]RegistryConfig `toml:"configs" json:"configs"`

cri/pkg/config/config.go

Line 155 in 8252e54

Auth *AuthConfig `toml:"auth" json:"auth"`

I've tried registry.auths and because registry.configs is a map[string]X that mirrors (sic.) mirrors, I tried the following too:

[plugins.cri.registry.configs."https://gcr.io".auth]

None of these work.

Is there some way that I can inspect the URL that it is try to make against the registry?

journalctl logs:

failed to resolve image "gcr.io/[redacted]/hello-world:latest": no available registry endpoint: unexpected status code https://gcr.io/v2/[redacted]/hello-world/manifests/latest: 401 Unauthorized"

But, if I take $(gcloud auth print-access-token) (which is what I'm using for the auth value), I can:

curl \
--request GET\
 --header "Authorization: Bearer $(gcloud auth print-access-token)" \
https://gcr.io/v2/[redacted]/hello-world/manifests/latest
{
   "schemaVersion": 2,
   "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
   "config": {
      "mediaType": "application/vnd.docker.container.image.v1+json",
      "size": 1510,
      "digest": "sha256:bf756fb1..."
   },
   "layers": [
      {
         "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
         "size": 2529,
         "digest": "sha256:0e03bdcc..."
      }
   ]
}

NOTE this is the same URL as returns 401 from containerd

mikebrow · 2020-05-16T18:58:17Z

restarting micro k8s... is that the parent for the instance of containerd? if not you need to restart containerd..

DazWilkin · 2020-05-16T19:01:17Z

IIUC, yes.

microk8s.stop && microk8s.start

yields:

stop of [... microk8s.daemon-containerd ...]

mikebrow · 2020-05-16T19:03:23Z

it may help to run containerd in debug (-D) but pretty sure we don't output the auth details for privacy reasons..

DazWilkin · 2020-05-16T19:25:03Z

IIUC dockerd leverages containerd and I'm able to find 2 containerd processes running (one for Docker and the other for MicroK8s). Unfortunately, while the MicroK8s configuration uses containerd.toml to define these registries and auths, I'm unable to correlate this to the containerd instance run by Docker which works with the GCR registry; if I docker login, that config is stored in config.json and doesn't appear to be reflected explicity in the containerd.toml.

Perhaps I should ping the comment you cited from MicroK8s?

There may be something particular to MicroK8s?

And/or, someone there may have tried with GCR.

mikebrow · 2020-05-16T19:29:52Z

Others reporting it does work..
k3s-io/k3s#1610

maybe I need to get a gcr account or try it against a service account

mikebrow · 2020-05-16T19:36:36Z

IIUC dockerd leverages containerd and I'm able to find 2 containerd processes running (one for Docker and the other for MicroK8s). Unfortunately, while the MicroK8s configuration uses containerd.toml to define these registries and auths, I'm unable to correlate this to the containerd instance run by Docker which works with the GCR registry; if I docker login, that config is stored in config.json and doesn't appear to be reflected explicity in the containerd.toml.

Perhaps I should ping the comment you cited from MicroK8s?

There may be something particular to MicroK8s?

And/or, someone there may have tried with GCR.

ah.. yeah while you can run with two containerd's you only need one containerd running... That should not cause you any problems...

containerd/cri uses the docker resolver code to make the connection.. hoping this is still just a config issue..

mikebrow · 2020-05-16T19:40:57Z

'{  "type": "service_account",  "project_id": "XXXX",  "private_key_id": "XXXX",  ...}'

note the single quotes due the to double quotes inside..

DazWilkin · 2020-05-16T19:53:53Z

Hmmm... the issue says that it doesn't work quoting this way (which is what I tried).

`{ "type": "service_account", ...}

It may be that I must escape the string:

"{ \"type\": \"service_account\", ...}

Still (!) I should not need to use a service account.

I should be able to authenticate using a access token.

I can create a service account for you to use with my registry?

mikebrow · 2020-05-16T19:54:42Z

yeah no.. I was just pointing out that these folks reported it working, in this fashion so there's hope.

DazWilkin · 2020-05-16T19:58:26Z

I'm going to try it on a different machine that's running MicroK8s too.

Please email me my handle at gmail dot com and I'll reply you the service account credentials.

I've given the account pull (and list only not push).

mikebrow · 2020-05-16T20:02:53Z

i'm going to go dig some holes in the back yard :-) will put gcr auth testing on the todo list for containerd/cri integration test unless you can also report it working, I need a week-end off, cheers

DazWilkin · 2020-05-16T20:04:55Z

Thanks for your help! Have fun! :-)

mikebrow · 2020-05-16T20:05:44Z

sure np it should be as simple as configure for gcr and run crictl to pull

mikebrow · 2020-05-16T20:05:55Z

ikr :-)

mikebrow · 2020-05-16T20:07:23Z

^ that way would eliminate anything microk8s is doing

DazWilkin · 2020-05-16T20:19:14Z

Escaping the password value was not the issue.

Abandoning for the weekend.

DazWilkin · 2020-05-17T00:10:06Z

I downloaded a standalone containerd (1.4.0-beta), applied the configurations I've been trying.

Same-same :-(

DazWilkin · 2020-05-18T17:55:21Z

OK.... Something curious.

I'm able to interact with ctr through MicroK8s and get more useful debugging:

sudo microk8s ctr --debug images pull gcr.io/[[PROJECT]]/hello-world:latest

I'm able to use this to pull e.g. public docker images successfully.

If I try to pull from GCR:

DEBU[0000] fetching                                      image="gcr.io/[[PROJECT]]/hello-world:latest"
DEBU[0000] resolving                                    
DEBU[0000] do request                                    request.headers=map[Accept:[application/vnd.docker.distribution.manifest.v2+json, application/vnd.docker.distribution.manifest.list.v2+json, application/vnd.oci.image.manifest.v1+json, application/vnd.oci.image.index.v1+json, *]] request.method=HEAD url="https://gcr.io/v2/[[PROJECT]]/hello-world/manifests/latest"
DEBU[0000] fetch response received                       response.headers=map[Accept-Ranges:[none] Alt-Svc:[h3-27=":443"; ma=2592000,h3-25=":443"; ma=2592000,h3-T050=":443"; ma=2592000,h3-Q050=":443"; ma=2592000,h3-Q049=":443"; ma=2592000,h3-Q048=":443"; ma=2592000,h3-Q046=":443"; ma=2592000,h3-Q043=":443"; ma=2592000,quic=":443"; ma=2592000; v="46,43"] Cache-Control:[private] Content-Type:[application/json] Date:[Mon, 18 May 2020 17:32:08 GMT] Docker-Distribution-Api-Version:[registry/2.0] Server:[Docker Registry] Vary:[Accept-Encoding] Www-Authenticate:[Bearer realm="https://gcr.io/v2/token",service="gcr.io",scope="repository:[[PROJECT]]/hello-world:pull"] X-Frame-Options:[SAMEORIGIN] X-Xss-Protection:[0]] status="401 Unauthorized" url="https://gcr.io/v2/[[PROJECT]]/hello-world/manifests/latest"
DEBU[0000] Unauthorized                                  header="Bearer realm="https://gcr.io/v2/token",service="gcr.io",scope="repository:[[PROJECT]]/hello-world:pull""
DEBU[0000] do request                                    request.headers=map[Accept:[application/vnd.docker.distribution.manifest.v2+json, application/vnd.docker.distribution.manifest.list.v2+json, application/vnd.oci.image.manifest.v1+json, application/vnd.oci.image.index.v1+json, *]] request.method=HEAD url="https://gcr.io/v2/[[PROJECT]]/hello-world/manifests/latest"
DEBU[0000] fetch response received                       response.headers=map[Accept-Ranges:[none] Alt-Svc:[h3-27=":443"; ma=2592000,h3-25=":443"; ma=2592000,h3-T050=":443"; ma=2592000,h3-Q050=":443"; ma=2592000,h3-Q049=":443"; ma=2592000,h3-Q048=":443"; ma=2592000,h3-Q046=":443"; ma=2592000,h3-Q043=":443"; ma=2592000,quic=":443"; ma=2592000; v="46,43"] Cache-Control:[private] Content-Type:[application/json] Date:[Mon, 18 May 2020 17:32:08 GMT] Docker-Distribution-Api-Version:[registry/2.0] Server:[Docker Registry] Vary:[Accept-Encoding] Www-Authenticate:[Bearer realm="https://gcr.io/v2/token",service="gcr.io",scope="repository:[[PROJECT]]/hello-world:pull"] X-Frame-Options:[SAMEORIGIN] X-Xss-Protection:[0]] status="401 Unauthorized" url="https://gcr.io/v2/[[PROJECT]]/hello-world/manifests/latest"
DEBU[0000] Unauthorized                                  header="Bearer realm="https://gcr.io/v2/token",service="gcr.io",scope="repository:[[PROJECT]]/hello-world:pull""
DEBU[0000] do request                                    request.headers=map[Accept:[application/vnd.docker.distribution.manifest.v2+json, application/vnd.docker.distribution.manifest.list.v2+json, application/vnd.oci.image.manifest.v1+json, application/vnd.oci.image.index.v1+json, *] Authorization:[Bearer REDACTED]] request.method=HEAD url="https://gcr.io/v2/[[PROJECT]]/hello-world/manifests/latest"
DEBU[0000] fetch response received                       response.headers=map[Accept-Ranges:[none] Alt-Svc:[h3-27=":443"; ma=2592000,h3-25=":443"; ma=2592000,h3-T050=":443"; ma=2592000,h3-Q050=":443"; ma=2592000,h3-Q049=":443"; ma=2592000,h3-Q048=":443"; ma=2592000,h3-Q046=":443"; ma=2592000,h3-Q043=":443"; ma=2592000,quic=":443"; ma=2592000; v="46,43"] Cache-Control:[private] Content-Type:[application/json] Date:[Mon, 18 May 2020 17:32:08 GMT] Docker-Distribution-Api-Version:[registry/2.0] Server:[Docker Registry] Vary:[Accept-Encoding] Www-Authenticate:[Bearer realm="https://gcr.io/v2/token",service="gcr.io",scope="repository:[[PROJECT]]/hello-world:pull"] X-Frame-Options:[SAMEORIGIN] X-Xss-Protection:[0]] status="401 Unauthorized" url="https://gcr.io/v2/[[PROJECT]]/hello-world/manifests/latest"
DEBU[0000] Unauthorized                                  header="Bearer realm="https://gcr.io/v2/token",service="gcr.io",scope="repository:[[PROJECT]]/hello-world:pull""
DEBU[0000] do request                                    request.headers=map[Accept:[application/vnd.docker.distribution.manifest.v2+json, application/vnd.docker.distribution.manifest.list.v2+json, application/vnd.oci.image.manifest.v1+json, application/vnd.oci.image.index.v1+json, *] Authorization:[Bearer REDACTED]] request.method=HEAD url="https://gcr.io/v2/[[PROJECT]]/hello-world/manifests/latest"
DEBU[0001] fetch response received                       response.headers=map[Accept-Ranges:[none] Alt-Svc:[h3-27=":443"; ma=2592000,h3-25=":443"; ma=2592000,h3-T050=":443"; ma=2592000,h3-Q050=":443"; ma=2592000,h3-Q049=":443"; ma=2592000,h3-Q048=":443"; ma=2592000,h3-Q046=":443"; ma=2592000,h3-Q043=":443"; ma=2592000,quic=":443"; ma=2592000; v="46,43"] Cache-Control:[private] Content-Type:[application/json] Date:[Mon, 18 May 2020 17:32:09 GMT] Docker-Distribution-Api-Version:[registry/2.0] Server:[Docker Registry] Vary:[Accept-Encoding] Www-Authenticate:[Bearer realm="https://gcr.io/v2/token",service="gcr.io",scope="repository:[[PROJECT]]/hello-world:pull"] X-Frame-Options:[SAMEORIGIN] X-Xss-Protection:[0]] status="401 Unauthorized" url="https://gcr.io/v2/[[PROJECT]]/hello-world/manifests/latest"
DEBU[0001] Unauthorized                                  header="Bearer realm="https://gcr.io/v2/token",service="gcr.io",scope="repository:[[PROJECT]]/hello-world:pull""
DEBU[0001] do request                                    request.headers=map[Accept:[application/vnd.docker.distribution.manifest.v2+json, application/vnd.docker.distribution.manifest.list.v2+json, application/vnd.oci.image.manifest.v1+json, application/vnd.oci.image.index.v1+json, *] Authorization:[Bearer REDACTED] request.method=HEAD url="https://gcr.io/v2/[[PROJECT]]/hello-world/manifests/latest"
DEBU[0001] fetch response received                       response.headers=map[Accept-Ranges:[none] Alt-Svc:[h3-27=":443"; ma=2592000,h3-25=":443"; ma=2592000,h3-T050=":443"; ma=2592000,h3-Q050=":443"; ma=2592000,h3-Q049=":443"; ma=2592000,h3-Q048=":443"; ma=2592000,h3-Q046=":443"; ma=2592000,h3-Q043=":443"; ma=2592000,quic=":443"; ma=2592000; v="46,43"] Cache-Control:[private] Content-Type:[application/json] Date:[Mon, 18 May 2020 17:32:09 GMT] Docker-Distribution-Api-Version:[registry/2.0] Server:[Docker Registry] Vary:[Accept-Encoding] Www-Authenticate:[Bearer realm="https://gcr.io/v2/token",service="gcr.io",scope="repository:[[PROJECT]]/hello-world:pull"] X-Frame-Options:[SAMEORIGIN] X-Xss-Protection:[0]] status="401 Unauthorized" url="https://gcr.io/v2/[[PROJECT]]/hello-world/manifests/latest"
DEBU[0001] Unauthorized                                  header="Bearer realm="https://gcr.io/v2/token",service="gcr.io",scope="repository:[[PROJECT]]/hello-world:pull""
DEBU[0001] do request                                    request.headers=map[Accept:[application/vnd.docker.distribution.manifest.v2+json, application/vnd.docker.distribution.manifest.list.v2+json, application/vnd.oci.image.manifest.v1+json, application/vnd.oci.image.index.v1+json, *] Authorization:[Bearer REDACTED]] request.method=HEAD url="https://gcr.io/v2/[[PROJECT]]/hello-world/manifests/latest"
DEBU[0001] fetch response received                       response.headers=map[Accept-Ranges:[none] Alt-Svc:[h3-27=":443"; ma=2592000,h3-25=":443"; ma=2592000,h3-T050=":443"; ma=2592000,h3-Q050=":443"; ma=2592000,h3-Q049=":443"; ma=2592000,h3-Q048=":443"; ma=2592000,h3-Q046=":443"; ma=2592000,h3-Q043=":443"; ma=2592000,quic=":443"; ma=2592000; v="46,43"] Cache-Control:[private] Content-Type:[application/json] Date:[Mon, 18 May 2020 17:32:09 GMT] Docker-Distribution-Api-Version:[registry/2.0] Server:[Docker Registry] Vary:[Accept-Encoding] Www-Authenticate:[Bearer realm="https://gcr.io/v2/token",service="gcr.io",scope="repository:[[PROJECT]]/hello-world:pull"] X-Frame-Options:[SAMEORIGIN] X-Xss-Protection:[0]] status="401 Unauthorized" url="https://gcr.io/v2/[[PROJECT]]/hello-world/manifests/latest"
ctr: failed to resolve reference "gcr.io/[[PROJECT]]/hello-world:latest": unexpected status code https://gcr.io/v2/[[PROJECT]]/hello-world/manifests/latest: 401 Unauthorized

I've redacted the tokens but they change each time which is OK except I should be able to base64 decode these and see something of the form:

_json_key:{ "type": "service_account", "project_id": .... }

But I get a bunch of binary data beginning 0x0090 0x03e6 -xfd83 ... (AJAD5v...)

DazWilkin · 2020-05-18T18:06:53Z

And:

sudo microk8s ctr version
Client:
  Version:  v1.2.5
  Revision: bb71b10fd8f58240ca47fbb579b9d1028eea7c84

Server:
  Version:  v1.2.5
  Revision: bb71b10fd8f58240ca47fbb579b9d1028eea7c84

mikebrow · 2020-05-18T21:10:10Z

it worked for me:

*** create gcp account with gcr, do all the steps to enable it to receive a
pushed image, including the download of a new _json_key (for a new
service account user..)... Then:

docker login -u _json_key -p "$(cat key.json)" gcr.io
docker push gcr.io/test-registry-277619/busybox
docker logout gcr.io

*** generate a single line for the key file generated by gcr:

jq -c . key.json

*** Edit the /etc/containerd/config.toml:

[plugins."io.containerd.grpc.v1.cri".registry]
[plugins."io.containerd.grpc.v1.cri".registry.mirrors]
  [plugins."io.containerd.grpc.v1.cri".registry.mirrors."docker.io"]
    endpoint = ["https://registry-1.docker.io"]
  [plugins.cri.registry.mirrors."gcr.io"]
    endpoint = ["https://gcr.io"]
    [plugins.cri.registry.auths]
[plugins.cri.registry.auths."https://gcr.io"]
  username = "_json_key"
  password = 'output from jq'

*** restart containerd
mike@mike-VirtualBox:~/go/src/github.com/kubernetes-sigs/cri-tools$ service containerd stop
mike@mike-VirtualBox:~/go/src/github.com/kubernetes-sigs/cri-tools$ service containerd start

*** pull the image with crictl:

mike@mike-VirtualBox:~/go/src/github.com/kubernetes-sigs/cri-tools$ sudo crictl pull gcr.io/test-registry-277619/busybox
DEBU[0000] get image connection                         
DEBU[0000] connect using endpoint 'unix:///run/containerd/containerd.sock' with '3s' timeout 
DEBU[0000] connected successfully using endpoint: unix:///run/containerd/containerd.sock 
DEBU[0000] PullImageRequest: &PullImageRequest{Image:&ImageSpec{Image:gcr.io/test-registry-277619/busybox,},Auth:nil,SandboxConfig:nil,} 
DEBU[0001] PullImageResponse: &PullImageResponse{ImageRef:sha256:78096d0a54788961ca68393e5f8038704b97d8af374249dc5c8faec1b8045e42,} 
Image is up to date for sha256:78096d0a54788961ca68393e5f8038704b97d8af374249dc5c8faec1b8045e42

mikebrow · 2020-05-18T21:10:46Z

mike@mike-VirtualBox:~/go/src/github.com/kubernetes-sigs/cri-tools$ sudo crictl rmi gcr.io/test-registry-277619/busybox
DEBU[0000] get image connection                         
DEBU[0000] connect using endpoint 'unix:///run/containerd/containerd.sock' with '3s' timeout 
DEBU[0000] connected successfully using endpoint: unix:///run/containerd/containerd.sock 
DEBU[0000] User specified image to be removed: gcr.io/test-registry-277619/busybox 
DEBU[0000] ImageStatusRequest: &ImageStatusRequest{Image:&ImageSpec{Image:gcr.io/test-registry-277619/busybox,},Verbose:false,} 
DEBU[0000] ImageStatusResponse: &ImageStatusResponse{Image:&Image{Id:sha256:78096d0a54788961ca68393e5f8038704b97d8af374249dc5c8faec1b8045e42,RepoTags:[gcr.io/test-registry-277619/busybox:latest],RepoDigests:[gcr.io/test-registry-277619/busybox@sha256:a7766145a775d39e53a713c75b6fd6d318740e70327aaa3ed5d09e0ef33fc3df],Size_:762868,Uid:nil,Username:,},Info:map[string]string{},} 
DEBU[0000] RemoveImageRequest: &RemoveImageRequest{Image:&ImageSpec{Image:gcr.io/test-registry-277619/busybox,},} 
DEBU[0000] RemoveImageResponse: &RemoveImageResponse{}  
Deleted: gcr.io/test-registry-277619/busybox:latest
mike@mike-VirtualBox:~/go/src/github.com/kubernetes-sigs/cri-tools$ sudo crictl pull gcr.io/test-registry-277619/busybox
DEBU[0000] get image connection                         
DEBU[0000] connect using endpoint 'unix:///run/containerd/containerd.sock' with '3s' timeout 
DEBU[0000] connected successfully using endpoint: unix:///run/containerd/containerd.sock 
DEBU[0000] PullImageRequest: &PullImageRequest{Image:&ImageSpec{Image:gcr.io/test-registry-277619/busybox,},Auth:nil,SandboxConfig:nil,} 
DEBU[0001] PullImageResponse: &PullImageResponse{ImageRef:sha256:78096d0a54788961ca68393e5f8038704b97d8af374249dc5c8faec1b8045e42,} 
Image is up to date for sha256:78096d0a54788961ca68393e5f8038704b97d8af374249dc5c8faec1b8045e42

mikebrow · 2020-05-18T21:13:23Z

@DazWilkin can you try with crictl? Let's try to minimize the surface area..

DazWilkin · 2020-05-18T22:28:24Z

Exciting!

I'd not actually tried using just username and password! Otherwise, the config matches.

I generally don't interact with these daemons directly so, my apologies for not being familiar with their tools either. I will work out how to use crictl

I'm walking my dog but will try this when home.

Thank you!

mikebrow · 2020-05-18T23:14:40Z

If you don't already have it... you could just clone this repo https://github.com/kubernetes-sigs/cri-tools then run make && make install. Otherwise I'd be surprised if the above config hack doesn't work for you.

mikebrow · 2020-05-18T23:17:01Z

my config for crictl.. in case you already have it and it's not configured to use containerd by default:

mike@mike-VirtualBox:~/go/src/github.com/kubernetes-sigs/cri-tools$ cat /etc/crictl.yaml
runtime-endpoint: unix:///run/containerd/containerd.sock
image-endpoint: ""
timeout: 3
debug: true

DazWilkin · 2020-05-18T23:56:22Z

Hmmm :-(

I'd been using:

[plugins.cri.registry.auths."gcr.io"]

NOTE this mirrors [plugins.cri.registry.mirrors."gcr.io"]

But, crictl works if this is:

[plugins.cri.registry.auths."https://gcr.io"]

And, per your repro, I only used the username and password values:

sudo crictl --runtime-endpoint=unix:/var/snap/microk8s/common/run/containerd.sock pull gcr.io/${PROJECT}/hello-world:latest
Image is up to date for sha256:bf756fb1ae65adf866bd8c456593cd24beb6a0a061dedf42b26a993176745f6b

However :-(

sudo microk8s ctr --debug images pull gcr.io/${PROJECT}/hello-world:latest

Still 401s :-(

DazWilkin · 2020-05-19T00:05:25Z

In other good news, this mechanism also works using:

username = "oauth2accesstoken"
password = "value from $(gcloud auth print-access-token)"

Which is preferable and further validates that this mechanism is good.

Very curiously (!) it also works in Kubernetes:

kubectl run hello-world --image=gcr.io/${PROJECT}/hello-world:latest
kubectl describe pod/hello-world
Successfully assigned default/hello-world to hp-pavilion
Pulling image "gcr.io/${PROJECT}/hello-world:latest"
Successfully pulled image "gcr.io/${PROJECT}/hello-world:latest"
Created container hello-world
Started container hello-world

So, my original problem is solved except I now don't understand why this doesn't work:

sudo microk8s ctr --debug images pull gcr.io/${PROJECT}/hello-world:latest

mikebrow · 2020-05-19T00:18:19Z

ctr isn't using containerd/cri so the CRI config won't modify how ctr tries to pull

mikebrow · 2020-05-19T00:20:43Z

containerd/cri is a plugin that operates via CRI service requests then calls to containerd to execute

crictl uses CRI services ..

containerd/CRI uses the config ^ above to generate the pull request with auth

mikebrow · 2020-05-19T00:23:22Z

i'm not familiar with microk8s code..

mikebrow · 2020-05-19T00:29:37Z

At some point we'll be refactoring the way we configure registry access, and that will include moving this config into containerd proper so ctr pulls would work with the same config.

DazWilkin · 2020-05-19T00:41:24Z

OK .... Well, thank you very much for your help!

To conclude:

The configuration is as you show
Albeit curiously while mirrors keys on gcr.io, auths keys on https://gcr.io
username and password are required (auth and identitytoken are not)
For service accounts _json_key and some variant of:
- cat key.json | jq -c .
- cat key.json | tr '\n' ' ' | sed 's|\"|\\"|'
For access tokens oauth2accesstoken and $(gcloud auth print-access-token)

It appears that it's not possible to use auth only and the Docker equivalents:

printf "_json_key:%s" $(cat thalia.json | jq -c .) \
| base64 --wrap=0

printf "oauth2accesstoken:%s" $(gcloud auth print-access-token) \
| base64 --wrap=0

DazWilkin · 2020-05-19T00:48:37Z

You've been very helpful and patient.

I appreciate the education on containerd and cri

I didn't realize the crictl is a tool recommended for Kubernetes node debugging and will now use it.

Thank you!

mikebrow · 2020-05-19T01:25:04Z

sure np.. I haven't played with GCP/GCR so learned some things. I'll draft some PRs maybe make this a bit easier, esp. the docs. FYI kublet uses crictl to integrate with dockershim, containerd, and cri-o. So a little more than just a node debugging tool. Not an end user tool.. by any means but supported nonetheless.

DazWilkin · 2020-05-19T01:34:40Z

Popping back from this rabbit hole...

I'm going to update the Krustlet team on how GCR can be used (as well as Azure Container Registry) for OCI images.

Krustlet doesn't support imagePullSecret and so it's necessary to configure CRI as shown here.

mikebrow · 2020-05-19T11:40:58Z

@DazWilkin Thx..
@SteveLasker just an FYI since ACR use pattern was mentioned above for pulling container images(artifacts etc.) with auth setup via containerd/cri+krustlet+microk8s

DazWilkin mentioned this issue May 18, 2020

Containerd fails to authenticate to a private Google Cloud Registry canonical/microk8s#990

Closed

mikebrow mentioned this issue May 22, 2020

add a registry auth tutorial #1490

Merged

Sh4d1 mentioned this issue Jan 8, 2021

Can't pull image while specifying user/pass in config containerd/containerd#4920

Closed

benmccown mentioned this issue Nov 4, 2021

Install crictl for CLI interaction with containerd awslabs/amazon-eks-ami#797

Closed

dmcgowan closed this as completed Mar 8, 2022

erkanerol mentioned this issue Jan 27, 2023

Support registry configuration for containerd giantswarm/cluster-cloud-director#79

Merged

4 tasks

Struggling to understand how to configure containerd.toml for a private registry #1482

Struggling to understand how to configure containerd.toml for a private registry #1482

Comments

DazWilkin commented May 15, 2020

DazWilkin commented May 15, 2020

mikebrow commented May 15, 2020

mikebrow commented May 15, 2020

DazWilkin commented May 15, 2020

mikebrow commented May 15, 2020

mikebrow commented May 15, 2020

mikebrow commented May 15, 2020

DazWilkin commented May 15, 2020 • edited Loading

DazWilkin commented May 15, 2020

DazWilkin commented May 16, 2020

mikebrow commented May 16, 2020

mikebrow commented May 16, 2020 • edited Loading

DazWilkin commented May 16, 2020

mikebrow commented May 16, 2020

DazWilkin commented May 16, 2020

mikebrow commented May 16, 2020

DazWilkin commented May 16, 2020

mikebrow commented May 16, 2020

mikebrow commented May 16, 2020

mikebrow commented May 16, 2020 • edited Loading

DazWilkin commented May 16, 2020

mikebrow commented May 16, 2020 • edited Loading

DazWilkin commented May 16, 2020

mikebrow commented May 16, 2020

DazWilkin commented May 16, 2020

mikebrow commented May 16, 2020

mikebrow commented May 16, 2020

mikebrow commented May 16, 2020

DazWilkin commented May 16, 2020

DazWilkin commented May 17, 2020

DazWilkin commented May 18, 2020 • edited Loading

DazWilkin commented May 18, 2020

mikebrow commented May 18, 2020 • edited Loading

mikebrow commented May 18, 2020

mikebrow commented May 18, 2020

DazWilkin commented May 18, 2020 • edited Loading

mikebrow commented May 18, 2020

mikebrow commented May 18, 2020

DazWilkin commented May 18, 2020

DazWilkin commented May 19, 2020

mikebrow commented May 19, 2020 • edited Loading

mikebrow commented May 19, 2020 • edited Loading

mikebrow commented May 19, 2020

mikebrow commented May 19, 2020

DazWilkin commented May 19, 2020

DazWilkin commented May 19, 2020 • edited Loading

mikebrow commented May 19, 2020

DazWilkin commented May 19, 2020

mikebrow commented May 19, 2020

DazWilkin commented May 15, 2020 •

edited

Loading

mikebrow commented May 16, 2020 •

edited

Loading

mikebrow commented May 16, 2020 •

edited

Loading

mikebrow commented May 16, 2020 •

edited

Loading

DazWilkin commented May 18, 2020 •

edited

Loading

mikebrow commented May 18, 2020 •

edited

Loading

DazWilkin commented May 18, 2020 •

edited

Loading

mikebrow commented May 19, 2020 •

edited

Loading

mikebrow commented May 19, 2020 •

edited

Loading

DazWilkin commented May 19, 2020 •

edited

Loading