Run abctl with 2 CPU & 4 GB of memory

[nataliekwong]

Today, abctl works on a machine with 4 or more CPU's and at least 8 GB of memory with the --low-resource-mode flag. You can pass the low resource mode flag when installing Airbyte with abctl:

abctl local install --low-resource-mode

You can see our suggested resources documentation here: https://docs.airbyte.com/using-airbyte/getting-started/oss-quickstart#suggested-resources

This equivalents to a t2.large (Amazon EC2 T2 Instances) using --low-resource-mode. It does not yet work on t2.medium.

This discussion aims to collect interest and feedback on enabling Airbyte to sufficiently run on 2 or more CPu's and 4 GB of memory, or a t2.medium instance.

[techieshark]

Very interested. Thanks for suggesting.

I'm evaluating Airbyte by running locally on an M1 with 16GB ram and actually struggling to get it up and running with the following settings (though at this point I'm not sure if I'm still running into resource issues or just some other local config issues).

[caileyfitzgerald]

Yes, this would be a great help! Since switching to abctl from docker compose in our staging environment, we are seeing significantly longer load times (up to hours) for even small amounts of data (20 records), whereas docker compose is able to load magnitudes higher in a fraction of the time.

[ngohuuduc]

We face the same thing.

[jediwarpraptor]

Is it the # of CPUs or the Memory that's the bottleneck. I'm contemplating standing this up on a 2-processor 16GB VM.

[et3rn4ls]

Is it possible to change to low resource mode after installation? I deployed it on a m6a.xlarge machine (4 vCPUs / 16 GB) but had problems with lack of resources (FailedScheduling). Now it is running on a m6a.2xlarge (8 vCPUs / 32GB) but it is not viable due to costs.

[colesnodgrass]

You could trying running the install command again passing the --low-resource-mode flag. An install over an existing install will attempt to upgrade the cluster in place.

abctl local install --low-resource-mode [any other flag originally provided]

If that fails, you can always uninstall and and install again. Your data should be persisted during the uninstall process.

abctl local uninstall && abctl local install --low-resource-mode [any other flags required]

[brothatru]

I'm running on AWS EC2 r7g.xlarge (4 cpus x 32 gb ram), but got hit with the error:

Unable to start the REPLICATION pod. This may be due to insufficient system resources. Please check available resources and try again.

Setting the --low-resource-mode works in my case.

What exactly does the --low-resource-mode flag do?

Are there any other ways to "fine tune" resources using environment variables? In my case to maximize my 4 cpus and 32 gb ram?

• ie. JOB_MAIN_CONTAINER_CPU_LIMIT or JOB_MAIN_CONTAINER_MEMORY_LIMIT?

Asking because I don't want "low resource mode" to limit to, for example 2 cpus x 8gb of ram, when my machine has much more.

Thanks in advance!

[umangrungta87]

Will Azure B2ms be supported in low-resource mode?
Based on airbytes specifications, it should work, but it keeps crashing repeatedly. How can I fix this? My usage isn't high.

[ngohuuduc]

You need to try. In my case, It was 4CPU and 12GB of RAM of my instance (shared with other services as well). I have to use low-resource mode.

[Rhiyo]

I've just migrated to abctl from docker compose.

I'm only dealing with small amount of data. About 7 connections. To make sure it doesn't hit limits it runs each connection one at a time.

With docker compose I was using a t3.medium. It usually ran fine but the CPU would get throttled and I'd get a winning it's hitting it's limit every other day when running a job.

With abctl I couldn't even install without hitting limits so I upgraded to a t3.large. We will see how that goes. I am using --low-resource-mode and also --no-browser to try and minimize the resources used. I had a few CPU warnings while experimenting but I will come back here to tell you how it's going in a few days.

I use --no-browser as I orchestrate my jobs externally and only bring up airbyte once a day and do all the syncs at once.

Preferablly I'd like a way that PyAirbyte can interact with the postgres airbyte db so I wouldn't have to bring up the airbyte server at all unless I am troubleshooting.