Document MAX_*_WORKERS in Workers & Jobs
#9191
Conversation
github-actions
bot
added
the
area/documentation
| ## Worker parallelization | ||
| Airbyte exposes the following environment variable to change the maximum number of each type of worker allowed to run in parallel. | ||
| Tweaking these values might help you run more jobs in parallel and increase the workload of your Airbyte instance: | ||
| * `MAX_SPEC_WORKERS`: Maximum number of *Spec* workers allowed to run in parallel. | ||
| * `MAX_CHECK_WORKERS`: Maximum number of *Check connection* workers allowed to run in parallel. | ||
| * `MAX_DISCOVERY_WORKERS`: Maximum number of *Discovery* workers allowed to run in parallel. | ||
| * `MAX_SYNC_WORKERS`: Maximum number of *Sync* workers allowed to run in parallel. | ||
|
|
||
| The current default value for these environment variables is currently set to **5**. |
There was a problem hiding this comment.
I don't think this is the right place for this. The Worker Parallelization only works for Kubernetes deployment. Also already have one section about the topic: https://docs.airbyte.io/deploying-airbyte/on-kubernetes#increasing-job-parallelism
There was a problem hiding this comment.
Are you sure they are only working for Kubernetes? When I check the call stack of these env var they are used to set a temporal WorkerOptions in airbyte-workers/src/main/java/io/airbyte/workers/WorkerApp.java:
private static WorkerOptions getWorkerOptions(final int max) {
return WorkerOptions.newBuilder()
.setMaxConcurrentActivityExecutionSize(max)
.build();
}I am not sure this temporal configuration is exclusive to K8S.
There was a problem hiding this comment.
Yeah, I am sure it works for K8s but I was not able to get it working in docker.
There was a problem hiding this comment.
@harshithmullapudi this should work in docker too. Why do you say you aren't able to get this working?
There was a problem hiding this comment.
In k8s I played with these parameters a bit earlier and could find the right config to run about 2000 syncs (in 4 hours, I could see 50 running at any point) with 50 parallel syncs (SUBMITTER_NUM_THREADS=50) with about 5 workers (with MAX_SYNC_WORKERS=20)
I tried doing the same with docker and I could only run 15 at max. Not sure if I am missing something
There was a problem hiding this comment.
do you remember what your submitter_num_threads and max_sync_workers vars were?
There was a problem hiding this comment.
Ah. I think we also need to increase the ports right? I gave 60 ports in K8s but didn't change it in docker.
There was a problem hiding this comment.
ah I think it's because the max sync worker variable is not actually passed into the worker container in the docker build config.
There was a problem hiding this comment.
Ah. I think we also need to increase the ports right? I gave 60 ports in K8s but didn't change it in docker.
Nope. This is only used in Kube. Docker doesn't need explicit port allocation.
There was a problem hiding this comment.
I'm going to fix this in #9209.
Will tag folks when I'm done. Apart from this change, what we have here is correct.
|
@marcosmarxm, as discussed with @davinchia above, these env var can be used both on docker and K8S deployment, and are now set in the docker-compose file. Can we merge this now? |
We often receive messages from users (e.g. this one) asking for help related to scaling and parallelization.
I found it helpful to add more details about
MAX_*_WORKERSenv var to the Worker & Jobs documentation.