[helm] Heating 429 Too Many Requests on orchestrator-repl-job pods

[AronsonDan]

Helm Chart Version

0.293.4

What step the error happened?

Upgrading the Platform or Helm Chart

Relevant information

During the upgrade to the new 0.293.4 helm chart im getting:

Failed to pull image "airbyte/container-orchestrator:0.63.8": failed to pull and unpack image "docker.io/airbyte/container-orchestrator:0.63.8": failed to copy: httpReadSeeker: failed open: unexpected status code https://registry1.docker.io/v2/airbyte/containerorchestrator/manifests/sha256:f2277e78343406ff9adddcb13b79078893e747ec3ef2353b7510c60c0a637e22: 429 Too Many Requests -Server message: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit

In my values, i set the following:

    jobs:
        main_container_image_pull_secret: "dockerhub-secret"
  worker:
    image:
      repository: 143653229561.dkr.ecr.us-east-1.amazonaws.com/docker-hub/airbyte/worker
    containerOrchestrator:
      image: 143653229561.dkr.ecr.us-east-1.amazonaws.com/docker-hub/airbyte/container-orchestrator:0.63.8
  workload-launcher:
    image:
      repository: 143653229561.dkr.ecr.us-east-1.amazonaws.com/docker-hub/airbyte/workload-launcher
    containerOrchestrator:
      image: 143653229561.dkr.ecr.us-east-1.amazonaws.com/docker-hub/airbyte/container-orchestrator:0.63.8

Im using ecr pull through cache so i wont have the docker server throttling issue.

When im looking at the airbyte-env config map, I see the following value:
CONTAINER_ORCHESTRATOR_IMAGE: 143653229561.dkr.ecr.us-east-1.amazonaws.com/docker-hub/airbyte/container-orchestrator:0.63.8

But when orchestrator-repl-job are starting they appear with the following image: image: airbyte/container-orchestrator:0.63.8

On top of that, when I set main_container_image_pull_secret: "dockerhub-secret" i would have expected the pull to be made using my docker hub credentials which is not happenning:

- name: JOB_KUBE_MAIN_CONTAINER_IMAGE_PULL_SECRET                                                                                                                  
   value: dockerhub-secret

Would love to get your help on that :-)

Relevant log output

No response

[AronsonDan]

Saw that line of code that might be related: https://github.com/airbytehq/airbyte-platform/blob/8f00c4184f6a90506caf3aea78c61c18ba090fad/airbyte-server/Tiltfile#L32

[marcosmarxm]

You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit

[AronsonDan]

@marcosmarxm We had that do to the fact that we had a separate Airbye deployment which caused a lot of docker pulling.

What I would really love is the ability to add a prefix to all of the docker images being pulled by Airbyte as the current solution would not tolerate significant scale.
Solving the issue using pull through cache would be significantly better solution.

BTW, i have authenticated as described in:

- name: JOB_KUBE_MAIN_CONTAINER_IMAGE_PULL_SECRET                                                                                                                  
   value: dockerhub-secret

[sivankumar86]

@marcosmarxm I have created a PR to resolve above issue. Could you please take a look on this ? it is a blocker. I have build custom image and applied as a workaround but, it is tedious to maintain during upgrade.

airbytehq/airbyte-platform#335

[sivankumar86]

issue : image secret is not passed to container-orchestrator pod creation API in worker code .
solution: get the secret from env and add it to pod creation API.

[octavia-squidington-iii]

At Airbyte, we seek to be clear about the project priorities and roadmap. This issue has not had any activity for 180 days, suggesting that it's not as critical as others. It's possible it has already been fixed. It is being marked as stale and will be closed in 20 days if there is no activity. To keep it open, please comment to let us know why it is important to you and if it is still reproducible on recent versions of Airbyte.

[octavia-squidington-iii]

This issue was closed because it has been inactive for 20 days since being marked as stale.