From 895f2bfe8402e61fbb10aec8ced51b12cda69c2c Mon Sep 17 00:00:00 2001 From: alafanechere Date: Wed, 29 Dec 2021 17:09:38 +0100 Subject: [PATCH] new Worker parallelization section in jobs.md --- docs/understanding-airbyte/jobs.md | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/docs/understanding-airbyte/jobs.md b/docs/understanding-airbyte/jobs.md index 3c7455ecf57f4..9c52a25c79a58 100644 --- a/docs/understanding-airbyte/jobs.md +++ b/docs/understanding-airbyte/jobs.md @@ -39,6 +39,16 @@ Note: When a source has passed all of its messages, the docker process should au See the [architecture overview](high-level-view.md) for more information about workers. +## Worker parallelization +Airbyte exposes the following environment variable to change the maximum number of each type of worker allowed to run in parallel. +Tweaking these values might help you run more jobs in parallel and increase the workload of your Airbyte instance: +* `MAX_SPEC_WORKERS`: Maximum number of *Spec* workers allowed to run in parallel. +* `MAX_CHECK_WORKERS`: Maximum number of *Check connection* workers allowed to run in parallel. +* `MAX_DISCOVERY_WORKERS`: Maximum number of *Discovery* workers allowed to run in parallel. +* `MAX_SYNC_WORKERS`: Maximum number of *Sync* workers allowed to run in parallel. + +The current default value for these environment variables is currently set to **5**. + ## Job State Machine Jobs in the worker follow the following state machine.