kubelet error: failed to create existing container: /kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort

[Rushi017]

i deployed offline k8s kubeadm cluster but if i check the status of kubelet i'm getting below error logs i'm not able to identify exact issue

5140 manager.go:1121] Failed to create existing container: /kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod192a343d_2f00_4cc3_b1df_be27655bf577.slice/crio-8c40aabba2266f6ac7920b300017f1b7d8ecaa72c7593add86d1b83f5898f1aa.scope: Error finding container 8c40aabba2266f6ac7920b300017f1b7d8ecaa72c7593add86d1b83f5898f1aa: Status 404 returned error can't find the container with id 8c40aabba2266f6ac7920b300017f1b7d8ecaa72c7593add86d1b83f5898f1aa
Feb 08 00:49:54 master.offline.com kubelet[5140]: E0208 00:49:54.111078 5140 manager.go:1121] Failed to create existing container: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74b5e214_da7c_4187_b19f_ada713292cc2.slice/crio-4c41adf48cfac60dc2d7f8ae8abd6827708e9fc7ed3ef54ac29b483503c39d6c.scope: Error finding container 4c41adf48cfac60dc2d7f8ae8abd6827708e9fc7ed3ef54ac29b483503c39d6c: Status 404 returned error can't find the container with id 4c41adf48cfac60dc2d7f8ae8abd6827708e9fc7ed3ef54ac29b483503c39d6c
Feb 08 00:49:54 master.offline.com kubelet[5140]: E0208 00:49:54.117293 5140 manager.go:1121] Failed to create existing container: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3aceea7086edcf09f2a2dbe56c846108.slice/crio-a53ed15bc62d9726159c176157ac5b907ccee82ecdb2db96f847d85bed89d1f2.scope: Error finding container a53ed15bc62d9726159c176157ac5b907ccee82ecdb2db96f847d85bed89d1f2: Status 404 returned error can't find the container with id a53ed15bc62d9726159c176157ac5b907ccee82ecdb2db96f847d85bed89d1f2
Feb 08 00:49:54 master.offline.com kubelet[5140]: E0208 00:49:54.137294 5140 manager.go:1121] Failed to create existing container: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb08664bee315fe9913117f472a519838.slice/crio-ebcd36c2498c887217c10c79e85b5b2a168a40b8c19de3794c23ca706737e449.scope: Error finding container ebcd36c2498c887217c10c79e85b5b2a168a40b8c19de3794c23ca706737e449: Status 404 returned error can't find the container with id ebcd36c2498c887217c10c79e85b5b2a168a40b8c19de3794c23ca706737e449

kubernetes version : 1.26.1
cri-o version: 1.25.1

[k8s-ci-robot]

This issue is currently awaiting triage.

If a SIG or subproject determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[pacoxu]

/sig node
Can this be reproduced in a general cluster? Would you share reproduce steps?

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle stale
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle rotten
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Reopen this issue with /reopen
• Mark this issue as fresh with /remove-lifecycle rotten
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close not-planned

[k8s-ci-robot]

@k8s-triage-robot: Closing this issue, marking it as "Not Planned".

In response to this:

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Reopen this issue with /reopen
• Mark this issue as fresh with /remove-lifecycle rotten
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close not-planned

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[zhurkin]

cri-o 1.28.1
kubernetes 1.27.6
crun 1.8.5
OS Debian 12.1 / RHEL 8

the logs are constantly writing

2023-10-04T16:59:00.156270+03:00 cicd-kub-control-01 kubelet[62668]: E1004 16:59:00.156208 62668 manager.go:1106] Failed to create existing container: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddcfc7140_67fa_4244_ac62_e24d2b9e13b7.slice/crio-ebf6907bcc59986eed15fe16c62a6033f1d13880c225b5f885b984c845b8b9e5: Error finding container ebf6907bcc59986eed15fe16c62a6033f1d13880c225b5f885b984c845b8b9e5: Status 404 returned error can't find the container with id ebf6907bcc59986eed15fe16c62a6033f1d13880c225b5f885b984c845b8b9e5 2023-10-04T17:00:00.151086+03:00 cicd-kub-control-01 kubelet[62668]: E1004 17:00:00.150621 62668 manager.go:1106] Failed to create existing container: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod580089d6e069d3234bc4d982905c6673.slice/crio-0f1080f80dc3ad60cbb35672bacab279dd769e65de7f91db2d2ed66e2e11d3a6: Error finding container 0f1080f80dc3ad60cbb35672bacab279dd769e65de7f91db2d2ed66e2e11d3a6: Status 404 returned error can't find the container with id 0f1080f80dc3ad60cbb35672bacab279dd769e65de7f91db2d2ed66e2e11d3a6 2023-10-04T17:00:00.152184+03:00 cicd-kub-control-01 kubelet[62668]: E1004 17:00:00.152089 62668 manager.go:1106] Failed to create existing container: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddcfc7140_67fa_4244_ac62_e24d2b9e13b7.slice/crio-ebf6907bcc59986eed15fe16c62a6033f1d13880c225b5f885b984c845b8b9e5: Error finding container ebf6907bcc59986eed15fe16c62a6033f1d13880c225b5f885b984c845b8b9e5: Status 404 returned error can't find the container with id ebf6907bcc59986eed15fe16c62a6033f1d13880c225b5f885b984c845b8b9e5 2023-10-04T17:00:00.152619+03:00 cicd-kub-control-01 kubelet[62668]: E1004 17:00:00.152574 62668 manager.go:1106] Failed to create existing container: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf60e48bc697ddaa09c2bace113ccb59b.slice/crio-bca7d8c214218c2ff7d1dfaa2c6eadc5e8593bd630bb085508916b7500965a11: Error finding container bca7d8c214218c2ff7d1dfaa2c6eadc5e8593bd630bb085508916b7500965a11: Status 404 returned error can't find the container with id bca7d8c214218c2ff7d1dfaa2c6eadc5e8593bd630bb085508916b7500965a11 2023-10-04T17:00:00.153007+03:00 cicd-kub-control-01 kubelet[62668]: E1004 17:00:00.152950 62668 manager.go:1106] Failed to create existing container: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod580089d6e069d3234bc4d982905c6673.slice/crio-77e41c1ad0a21b896eb9df24267ac0c55185ce15336298c531d36f55a490f8b5: Error finding container 77e41c1ad0a21b896eb9df24267ac0c55185ce15336298c531d36f55a490f8b5: Status 404 returned error can't find the container with id 77e41c1ad0a21b896eb9df24267ac0c55185ce15336298c531d36f55a490f8b5 2023-10-04T17:00:00.153315+03:00 cicd-kub-control-01 kubelet[62668]: E1004 17:00:00.153261 62668 manager.go:1106] Failed to create existing container: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3d900f7fd7a358fec54765393bacd41.slice/crio-22a281781306a55505d15d06a7c172fbb97e90175e910ae049d2fba03204ce3c: Error finding container 22a281781306a55505d15d06a7c172fbb97e90175e910ae049d2fba03204ce3c: Status 404 returned error can't find the container with id 22a281781306a55505d15d06a7c172fbb97e90175e910ae049d2fba03204ce3c 2023-10-04T17:00:00.153649+03:00 cicd-kub-control-01 kubelet[62668]: E1004 17:00:00.153569 62668 manager.go:1106] Failed to create existing container: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod100eb816229ddc0fd2a10ad278565d23.slice/crio-a35162a2d75460edf8ed8a347aa6d4a67bda094b92819c1d93622d7c67135fb0: Error finding container a35162a2d75460edf8ed8a347aa6d4a67bda094b92819c1d93622d7c67135fb0: Status 404 returned error can't find the container with id a35162a2d75460edf8ed8a347aa6d4a67bda094b92819c1d93622d7c67135fb0 2023-10-04T17:00:00.153951+03:00 cicd-kub-control-01 kubelet[62668]: E1004 17:00:00.153883 62668 manager.go:1106] Failed to create existing container: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb51294dc_3ae8_4fc7_9b54_ecd66a74bb10.slice/crio-44151e525161cec3906e0780cee19d9a9949cd7594485db2a005b4791ce54d03: Error finding container 44151e525161cec3906e0780cee19d9a9949cd7594485db2a005b4791ce54d03: Status 404 returned error can't find the container with id 44151e525161cec3906e0780cee19d9a9949cd7594485db2a005b4791ce54d0