Low-code SDK: Error when adding backoff strategy

[delenamalan]

Environment

• Airbyte version: 0.1.99
• OS Version / Instance: macOS Montery
• Deployment: Docker
• Source Connector and version: Any
• Destination Connector and version: Not applicable
• Step where error happened: Update connector

Current Behavior

When I add a backoff strategy to the schema of a low-code SDK connector, I get an error like this when running python main.py read --config secrets/config.json --catalog integration_tests/configured_catalog.json --debug:

failed to create object of type <class 'airbyte_cdk.sources.declarative.requesters.error_handlers.backoff_strategies.wait_until_time_from_header_backoff_strategy.WaitUntilTimeFromHeaderBackoffStrategy'> because __init__() got an unexpected keyword argument 'options'

Expected Behavior

I should be able to use one of the backoff strategies without getting an exception.

Logs

Full logs

{"type": "LOG", "log": {"level": "FATAL", "message": "failed to create object of type <class 'airbyte_cdk.sources.declarative.requesters.error_handlers.backoff_strategies.wait_until_time_from_header_backoff_strategy.WaitUntilTimeFromHeaderBackoffStrategy'> because __init__() got an unexpected keyword argument 'options'\nTraceback (most recent call last):\n  File \"/Users/delena.malan/Workspace/airbyte/airbyte-integrations/connectors/source-greenhouse/.venv/lib/python3.9/site-packages/airbyte_cdk/sources/declarative/create_partial.py\", line 54, in newfunc\n    ret = func(*args, *fargs, **dynamic_args)\nTypeError: __init__() got an unexpected keyword argument 'options'\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n  File \"/Users/delena.malan/Workspace/airbyte/airbyte-integrations/connectors/source-greenhouse/main.py\", line 13, in <module>\n    launch(source, sys.argv[1:])\n  File \"/Users/delena.malan/Workspace/airbyte/airbyte-integrations/connectors/source-greenhouse/.venv/lib/python3.9/site-packages/airbyte_cdk/entrypoint.py\", line 123, in launch\n    for message in source_entrypoint.run(parsed_args):\n  File \"/Users/delena.malan/Workspace/airbyte/airbyte-integrations/connectors/source-greenhouse/.venv/lib/python3.9/site-packages/airbyte_cdk/entrypoint.py\", line 114, in run\n    for message in generator:\n  File \"/Users/delena.malan/Workspace/airbyte/airbyte-integrations/connectors/source-greenhouse/.venv/lib/python3.9/site-packages/airbyte_cdk/sources/abstract_source.py\", line 100, in read\n    stream_instances = {s.name: s for s in self.streams(config)}\n  File \"/Users/delena.malan/Workspace/airbyte/airbyte-integrations/connectors/source-greenhouse/.venv/lib/python3.9/site-packages/airbyte_cdk/sources/declarative/yaml_declarative_source.py\", line 66, in streams\n    source_streams = [self._factory.create_component(stream_config, config, True)() for stream_config in self._stream_configs()]\n  File \"/Users/delena.malan/Workspace/airbyte/airbyte-integrations/connectors/source-greenhouse/.venv/lib/python3.9/site-packages/airbyte_cdk/sources/declarative/yaml_declarative_source.py\", line 66, in <listcomp>\n    source_streams = [self._factory.create_component(stream_config, config, True)() for stream_config in self._stream_configs()]\n  File \"/Users/delena.malan/Workspace/airbyte/airbyte-integrations/connectors/source-greenhouse/.venv/lib/python3.9/site-packages/airbyte_cdk/sources/declarative/parsers/factory.py\", line 129, in create_component\n    return self.build(\n  File \"/Users/delena.malan/Workspace/airbyte/airbyte-integrations/connectors/source-greenhouse/.venv/lib/python3.9/site-packages/airbyte_cdk/sources/declarative/parsers/factory.py\", line 146, in build\n    updated_kwargs = {k: self._create_subcomponent(k, v, kwargs, config, class_, instantiate) for k, v in kwargs.items()}\n  File \"/Users/delena.malan/Workspace/airbyte/airbyte-integrations/connectors/source-greenhouse/.venv/lib/python3.9/site-packages/airbyte_cdk/sources/declarative/parsers/factory.py\", line 146, in <dictcomp>\n    updated_kwargs = {k: self._create_subcomponent(k, v, kwargs, config, class_, instantiate) for k, v in kwargs.items()}\n  File \"/Users/delena.malan/Workspace/airbyte/airbyte-integrations/connectors/source-greenhouse/.venv/lib/python3.9/site-packages/airbyte_cdk/sources/declarative/parsers/factory.py\", line 200, in _create_subcomponent\n    return self.create_component(definition, config, instantiate)()\n  File \"/Users/delena.malan/Workspace/airbyte/airbyte-integrations/connectors/source-greenhouse/.venv/lib/python3.9/site-packages/airbyte_cdk/sources/declarative/parsers/factory.py\", line 129, in create_component\n    return self.build(\n  File \"/Users/delena.malan/Workspace/airbyte/airbyte-integrations/connectors/source-greenhouse/.venv/lib/python3.9/site-packages/airbyte_cdk/sources/declarative/parsers/factory.py\", line 146, in build\n    updated_kwargs = {k: self._create_subcomponent(k, v, kwargs, config, class_, instantiate) for k, v in kwargs.items()}\n  File \"/Users/delena.malan/Workspace/airbyte/airbyte-integrations/connectors/source-greenhouse/.venv/lib/python3.9/site-packages/airbyte_cdk/sources/declarative/parsers/factory.py\", line 146, in <dictcomp>\n    updated_kwargs = {k: self._create_subcomponent(k, v, kwargs, config, class_, instantiate) for k, v in kwargs.items()}\n  File \"/Users/delena.malan/Workspace/airbyte/airbyte-integrations/connectors/source-greenhouse/.venv/lib/python3.9/site-packages/airbyte_cdk/sources/declarative/parsers/factory.py\", line 200, in _create_subcomponent\n    return self.create_component(definition, config, instantiate)()\n  File \"/Users/delena.malan/Workspace/airbyte/airbyte-integrations/connectors/source-greenhouse/.venv/lib/python3.9/site-packages/airbyte_cdk/sources/declarative/parsers/factory.py\", line 129, in create_component\n    return self.build(\n  File \"/Users/delena.malan/Workspace/airbyte/airbyte-integrations/connectors/source-greenhouse/.venv/lib/python3.9/site-packages/airbyte_cdk/sources/declarative/parsers/factory.py\", line 146, in build\n    updated_kwargs = {k: self._create_subcomponent(k, v, kwargs, config, class_, instantiate) for k, v in kwargs.items()}\n  File \"/Users/delena.malan/Workspace/airbyte/airbyte-integrations/connectors/source-greenhouse/.venv/lib/python3.9/site-packages/airbyte_cdk/sources/declarative/parsers/factory.py\", line 146, in <dictcomp>\n    updated_kwargs = {k: self._create_subcomponent(k, v, kwargs, config, class_, instantiate) for k, v in kwargs.items()}\n  File \"/Users/delena.malan/Workspace/airbyte/airbyte-integrations/connectors/source-greenhouse/.venv/lib/python3.9/site-packages/airbyte_cdk/sources/declarative/parsers/factory.py\", line 209, in _create_subcomponent\n    return self.create_component(definition, config, instantiate)()\n  File \"/Users/delena.malan/Workspace/airbyte/airbyte-integrations/connectors/source-greenhouse/.venv/lib/python3.9/site-packages/airbyte_cdk/sources/declarative/parsers/factory.py\", line 129, in create_component\n    return self.build(\n  File \"/Users/delena.malan/Workspace/airbyte/airbyte-integrations/connectors/source-greenhouse/.venv/lib/python3.9/site-packages/airbyte_cdk/sources/declarative/parsers/factory.py\", line 146, in build\n    updated_kwargs = {k: self._create_subcomponent(k, v, kwargs, config, class_, instantiate) for k, v in kwargs.items()}\n  File \"/Users/delena.malan/Workspace/airbyte/airbyte-integrations/connectors/source-greenhouse/.venv/lib/python3.9/site-packages/airbyte_cdk/sources/declarative/parsers/factory.py\", line 146, in <dictcomp>\n    updated_kwargs = {k: self._create_subcomponent(k, v, kwargs, config, class_, instantiate) for k, v in kwargs.items()}\n  File \"/Users/delena.malan/Workspace/airbyte/airbyte-integrations/connectors/source-greenhouse/.venv/lib/python3.9/site-packages/airbyte_cdk/sources/declarative/parsers/factory.py\", line 213, in _create_subcomponent\n    return [\n  File \"/Users/delena.malan/Workspace/airbyte/airbyte-integrations/connectors/source-greenhouse/.venv/lib/python3.9/site-packages/airbyte_cdk/sources/declarative/parsers/factory.py\", line 214, in <listcomp>\n    self._create_subcomponent(\n  File \"/Users/delena.malan/Workspace/airbyte/airbyte-integrations/connectors/source-greenhouse/.venv/lib/python3.9/site-packages/airbyte_cdk/sources/declarative/parsers/factory.py\", line 200, in _create_subcomponent\n    return self.create_component(definition, config, instantiate)()\n  File \"/Users/delena.malan/Workspace/airbyte/airbyte-integrations/connectors/source-greenhouse/.venv/lib/python3.9/site-packages/airbyte_cdk/sources/declarative/create_partial.py\", line 56, in newfunc\n    raise Exception(f\"failed to create object of type {func} because {e}\")\nException: failed to create object of type <class 'airbyte_cdk.sources.declarative.requesters.error_handlers.backoff_strategies.wait_until_time_from_header_backoff_strategy.WaitUntilTimeFromHeaderBackoffStrategy'> because __init__() got an unexpected keyword argument 'options'"}}

Steps to Reproduce

1. Open the YAML file for the Greenhouse source, for example.
2. Replace the error_handler section with the following to use a backoff strategy:

    error_handler:
      backoff_strategies:
        - type: "WaitTimeFromHeader"
          header: "myheader"

4. Run python main.py read --config secrets/config.json --catalog integration_tests/configured_catalog.json --debug.

[misteryeo]

@delenamalan Thanks for the feedback! We'll take a look at this and let you know next steps. Feel free to share future questions/issues on our Slack channel too!

cc: @girarda @brianjlai

[brianjlai]

tagging as P2, but active PR in flight here: #18053

[delenamalan]

Hey @brianjlai, could this issue be labeled as "hacktober" as well please? 🙏