How to pull image from private registry?

[hillbun]

Thanks for helping us to improve k3s! We welcome all bug reports. Please fill out each area of the template so we can better help you. You can delete this message portion of the bug report.

Version:
Provide the output from k3s -v and provide the flags used to install or run k3s server.

k3s version v0.10.0 (f9888ca)

Describe the bug
A clear and concise description of what the bug is.

I config /var/lib/rancher/k3s/agent/etc/containerd/config.toml.tmpl adding

[plugins.cri.registry.mirrors]
[plugins.cri.registry.mirrors."docker.xxx.cn"]
endpoint = ["http://docker.xxx.cn:5000"]

[plugins.cri.registry.configs."docker.xxx.cn".auth]
username = "xxxx"
password = "xxxxxxx"

when I pull

k3s crictl pull docker.xxx.cn:5000/maxfaith/miop_ui:development

error returned:

FATA[2019-10-30T14:23:39.122757012+08:00] pulling image failed: rpc error: code = Unknown desc = failed to pull and unpack image "docker.xxx.cn:5000/maxfaith/miop_ui:development": failed to resolve reference "docker.xxx.cn:5000/maxfaith/miop_ui:development": failed to do request: Head https://docker.xxx.cn:5000/v2/maxfaith/miop_ui/manifests/development: x509: certificate has expired or is not yet valid

To Reproduce
Steps to reproduce the behavior:

Expected behavior
A clear and concise description of what you expected to happen.

Actual behavior
A clear and concise description of what actually happened.

Additional context
Add any other context about the problem here.

[xiaods]

hi, have a try this: k3s crictl pull --creds USERNAME[:PASSWORD]

[carlosrmendes]

hi configured my private registry on /etc/rancher/k3s/registries.yaml file and I cannot pull my image.
I'm getting this error running version 0.10.2:
Failed to pull image "repo_address/image_name:tag": rpc error: code = NotFound desc = failed to pull and unpack image "repo_address/image_name:tag": failed to resolve reference "repo_address/image_name:tag": repo_address/image_name:tag: not found

[xiaods]

@carlosmkb i suggest use cmd to pull image again.

[hillbun]

hi, have a try this: k3s crictl pull --creds USERNAME[:PASSWORD]

I tried this, same error

FATA[2019-10-30T14:23:39.122757012+08:00] pulling image failed: rpc error: code = Unknown desc = failed to pull and unpack image "docker.xxx.cn:5000/maxfaith/miop_ui:development": failed to resolve reference "docker.xxx.cn:5000/maxfaith/miop_ui:development": failed to do request: Head https://docker.xxx.cn:5000/v2/maxfaith/miop_ui/manifests/development: x509: certificate has expired or is not yet valid

[hillbun]

@xiaods

it seems this config not effected

[plugins.cri.registry.mirrors]
[plugins.cri.registry.mirrors."docker.xxx.cn"]
endpoint = ["http://docker.xxx.cn:5000"]

[xiaods]

x509: certificate has expired or is not yet valid， it maybe your ntp time is not sync on master, and client server. please double check the situation.

[xiaods]

@hillbun have you try it

[hillbun]

@xiaods

mostly it is not time problem.

My private registry is insecure.

any similar configureation in docker to solve this problem?
"insecure-registries": ["docker.xxx.cn:5000"]

[hillbun]

[plugins.cri.registry.mirrors]
[plugins.cri.registry.mirrors."docker.xxx.cn:5000"]
endpoint = ["http://docker.xxx.cn:5000"]

[plugins.cri.registry.configs."docker.xxx.cn:5000".auth]
username = "myusername"
password = "mypassword"

crictl pull docker.xxx.cn:5000/maxfaith/miop_ui:development
FATA[2019-11-07T15:29:08.826492009+08:00] pulling image failed: rpc error: code = Unknown desc = failed to pull and unpack image "docker.xxx.cn:5000/maxfaith/miop_ui:development": failed to resolve reference "docker.xxx.cn:5000/maxfaith/miop_ui:development": failed to do request: Head http://docker.xxx.cn:5000/v2/maxfaith/miop_ui/manifests/development: net/http: HTTP/1.x transport connection broken: malformed HTTP response "\x15\x03\x01\x00\x02\x02"

the errir occupied

[hillbun]

also, when k3s yalm define imagePullSecrets, it can not pull image either.

[xiaods]

@hillbun sorry for your waiting. i am on a busy case , i will go to setup a environment go test on your case asap.

[xiaods]

@hillbun go through k3s issue list. found a potential issue: #145 don't know it cause the bug. it should be check.

[xiaods]

https://www.civo.com/learn/set-up-a-private-docker-registry-with-tls-on-kubernetes FYI @hillbun

[hillbun]

@xiaods

As I mentioned, my registry is NON tls

[ysolis]

is this a duplicate of #502 ?

well, in any case i don't have any problems to use a private registry
#502 (comment)

[stone-wlg]

my k3s version is v1.0.0. i have setup docker registry without https and i use --private-registry registries.yaml.
if i setup k3s with --docker, k3s can use private registry.
if i setup k3s without --docker, k3s never use private registry.

[xiaods]

the --docker options is what purpose? @stone-wlg please give more insight to me. curious for the situation that.

[stone-wlg]

the --docker options is what purpose? @stone-wlg please give more insight to me. curious for the situation that.

https://rancher.com/docs/k3s/latest/en/installation/install-options/
--docker (agent/runtime) Use docker instead of containerd

[riker09]

I can confirm that the default implementation with containerd does not support GitHub Package Registry (GPR) with imagePullSecrets. It seems that imagePullSecrets are not supported by k3s, see this thread on StackOverflow.

However, when running the k3s cluster with the --docker option everything works fine. So the issue must be in containerd, not k3s. 🤔 🤷‍♂️

[ysolis]

I can confirm that the default implementation with containerd does not support GitHub Package Registry (GPR) with imagePullSecrets. It seems that imagePullSecrets are not supported by k3s, see this thread on StackOverflow.

However, when running the k3s cluster with the --docker option everything works fine. So the issue must be in containerd, not k3s. thinking man_shrugging

@riker09 @stone-wlg the problem is apparently only related to Github registry. I have an image in a Gitlab Registry in a personal private repository in gitlab.com, I have deployed k3s 1.0 with default options (servicelb, traefik and containerd), created the secret with kubectl create secret ..., defined the deploy token needed in gitlab to access the private repo/registry ... and the pod was deployed. Please confirm if this is the case. If you need the info, I follow the steps described in [1] and [2] using the imagePullSecrets option in the deployment yaml file.

[1] https://dimsolution.com/blog/2018-07-06/kubernetes-using-a-private-registry-like-gitlab/
[2] https://blog.zedroot.org/2019/01/21/gitlab-ci-kubernetes-pull-a-private-image-from-a-k8s-pod/

[erikwilson]

It looks like an issue with the Github Package Registry or containerd (or both) containerd/containerd#3291

[webees]

Similar help is needed.

containerd/containerd#4452

[ramran-r]

I am trying to pull images from local docker registry by keeping 'pullSecrets:' in deployment yaml, however it is always referring to 'registries.yaml' ( which has TLS Cert & auth details ) . I see some of them succeeded to pull the images using 'pullSecrets'. Could you pls. share the steps followed or is it still known issues with K3S.

[caroline-suse-rancher]

Closing this due to age