Set resource limits for a connector definition

[cgardens]

Tell us about the problem you're trying to solve

Since connectors now respect resource limits (when configured), we want to be able to tune those resource limits by connector. For example, we want there to be a default resource limit for connectors that is relatively low (~200-300MB), but some connectors, we want to allow to use more resource. e.g pokeapi source uses very little memory, while in many cases the bigquery destination needs more.

Describe the solution you’d like

Add the ability to set resource limits at the connector definition level.

Execution:

• add this to the actor_definitions table
• pipe it through to the worker
• add it to the api (in the future we can add it in the UI)

Note: After this change, there will be 3 places connector resources can be set. As an environment variable (worker configs), for a connector definition and for a connection. We should respect the limits in this order: connection > connector definition > worker configs.

Acceptance Criteria

• As an airbyter engineer, I am able to set the resources limits (cpu and memory) for a connector definition via API and the DB directly.
• This configuration should override the default connector resource limits but resource limits specified at the connection leve.

[pmossman]

Backlog Grooming notes:

• worker that is running should just use whatever the appropriate resource limit according to the hierarchy
• we won't require this to be set for a connector definition (at least to start with)