Docker & registries.yaml does not work for GCR

[wstarzak]

Environmental Info:
K3s Version: v1.20.8+k3s1 (5d0a12a)

Node(s) CPU architecture, OS, and Version:
Linux ubuntu 5.4.0-77-generic #86-Ubuntu SMP Thu Jun 17 02:35:03 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Docker version 20.10.7, build f0df350

Cluster Configuration: single node install

Describe the bug:

K3S does not apply docker private (Google Cloud Registry - GCR) credentials when put to registries.yaml

Steps To Reproduce:

• K3S installed with docker runtime
• Created /etc/rancher/k3s/registries.yaml

Filled in with:

mirrors:
  eu.gcr.io:
    endpoint:
      - "https://eu.gcr.io"
configs:
  eu.gcr.io:
    auth:
      username: _json_key
      password: |
        {
          "type": "service_account",
          "project_id": "",
          "private_key_id": "",
          "private_key": "-----BEGIN PRIVATE KEY-----\\/+z+\nf/+/>
          "client_email": "nt.com",
          "client_id": "",
          "auth_uri": "https://accounts.google.com/o/oauth2/auth",
          "token_uri": "https://oauth2.googleapis.com/token",
          "auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs",
          "client_x509_cert_url": ""
        }

(tried also escaping the new lines etc.)

Expected behavior:

It works

Actual behavior:

k3s crictl -D pull eu.gcr.io/test/test:latest

Gives an error of:

FATA[2021-07-15T10:23:10.690690133Z] pulling image: rpc error: code = Unknown desc = Error response from daemon: unauthorized: You don't have the needed permissions to perform this operation, and you may have invalid credentials. To authenticate your request, follow the steps in: https://cloud.google.com/container-registry/docs/advanced-authentication

Additional context / logs:

Some things such as project names has been redacted. SIlmilar issues #1610 #2367

[brandond]

registries.yaml does not work with Docker. The first line on the private registry docs page says:

Containerd can be configured to connect to private registries and use them to pull private images on the node.

[wstarzak]

@brandond And that leads me to the question of why after using docker login i can pull images using docker pull but K3S is not able to authenticate

[brandond]

Because docker login stores credentials in your home directory, and these credentials are passed into the docker daemon when you run docker pull from your shell. K3s does not have access to these credentials. If you want to use authenticated image pulls with Docker, you probably want to use Kubernetes Image Pull Secrets.