🐛 Source Kustomer: fix KeyError: 'user_agent' when retrieving schema

[ivansabik]

What

Currently the Kustomer source connector is broken (see #14965).

With current master branch:

With this PR:

How

The User Agent field is required in the config.json, it was removed in a previous PR but the documentation does not state why. This parameter is used when initializing the tap:

https://github.com/singer-io/tap-kustomer/blob/b56a2cc3646d57b8c1eb8876a335073a8968fd00/tap_kustomer/__init__.py#L36-L37

Background:

• Kustomer source fails loading schema with KeyError: 'user_agent'" #14965
• Kustomer source connector: investigate "user_agent" deprecation #8697
• 🎉 Source Kustomer: Clarify setup fields #8738

🚨 User Impact 🚨

The connector configuration now requires to specify the User Agent to be used for Kustomer API.

Pre-merge Checklist

Expand the relevant checklist and delete the others.

Updating a connector

Community member or Airbyter

• Grant edit access to maintainers (instructions)
• Secrets in the connector's spec are annotated with airbyte_secret
• Unit & integration tests added and passing. Community members, please provide proof of success locally e.g: screenshot or copy-paste unit, integration, and acceptance test output. To run acceptance tests for a Python connector, follow instructions in the README. For java connectors run ./gradlew :airbyte-integrations:connectors:<name>:integrationTest.
• Code reviews completed
• Documentation updated
  • Connector's README.md
  • Connector's bootstrap.md. See description and examples
  • Changelog updated in docs/integrations/<source or destination>/<name>.md including changelog. See changelog example
• PR name follows PR naming conventions

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

• Create a non-forked branch based on this PR and test the below items on it
• Build is successful
• If new credentials are required for use in CI, add them to GSM. Instructions.
• /test connector=connectors/<name> command is passing
• New Connector version released on Dockerhub and connector version bumped by running the /publish command described here

[CLAassistant]

All committers have signed the CLA.

[sajarin]

/test connector=connectors/source-kustomer-singer

🕑 connectors/source-kustomer-singer https://github.com/airbytehq/airbyte/actions/runs/2776491107
❌ connectors/source-kustomer-singer https://github.com/airbytehq/airbyte/actions/runs/2776491107
🐛

[ivansabik]

Error does not seem to be related, but not sure how to make integration tests pass successfully:

ci_common_utils installed: attrs==22.1.0,certifi==2022.6.15,cffi==1.15.1,charset-normalizer==2.1.0,-e git+https://github.com/Solofundsnyc/airbyte@1fa3a4ca7149670c69934ad30f265d8e34ee1c74#egg=ci_common_utils&subdirectory=tools/ci_common_utils,cryptography==37.0.4,flake8==5.0.2,idna==3.3,iniconfig==1.1.1,mccabe==0.7.0,packaging==21.3,pluggy==1.0.0,py==1.11.0,pycodestyle==2.9.0,pycparser==2.21,pyflakes==2.5.0,PyJWT==2.3.0,pyparsing==3.0.9,pyproject-flake8==0.0.1a4,pytest==6.2.5,requests==2.28.1,requests-mock==1.9.3,six==1.16.0,toml==0.10.2,tomli==2.0.1,urllib3==1.26.11
ci_common_utils run-test-pre: PYTHONHASHSEED='3470393811'
ci_common_utils run-test: commands[0] | pflake8 --config /actions-runner/_work/airbyte/airbyte/tools/../pyproject.toml /actions-runner/_work/airbyte/airbyte/tools/ci_common_utils
Traceback (most recent call last):
  File "/actions-runner/_work/airbyte/airbyte/tools/.tox/ci_common_utils/bin/pflake8", line 5, in <module>
    from pflake8.__main__ import main
  File "/actions-runner/_work/airbyte/airbyte/tools/.tox/ci_common_utils/lib/python3.9/site-packages/pflake8/__init__.py", line 54, in <module>
    class ModifiedConfigFileFinder(flake8.options.config.ConfigFileFinder):
AttributeError: module 'flake8.options.config' has no attribute 'ConfigFileFinder'

[sajarin]

/test connector=connectors/source-kustomer-singer

🕑 connectors/source-kustomer-singer https://github.com/airbytehq/airbyte/actions/runs/2777016235
❌ connectors/source-kustomer-singer https://github.com/airbytehq/airbyte/actions/runs/2777016235
🐛

[sajarin]

Hi @ivansabik, could you pull the latest changes from master? We recently made some changes to the github actions workflow.

[sajarin]

/test connector=connectors/source-kustomer-singer

🕑 connectors/source-kustomer-singer https://github.com/airbytehq/airbyte/actions/runs/2783727017
✅ connectors/source-kustomer-singer https://github.com/airbytehq/airbyte/actions/runs/2783727017
No Python unittests run

Build Passed

Test summary info:

All Passed

[sajarin]

LGTM! Thanks for your contribution!

[sajarin]

/publish connector=connectors/source-kustomer-singer

🕑 Publishing the following connectors:
connectors/source-kustomer-singer
https://github.com/airbytehq/airbyte/actions/runs/2784880942

Connector	Did it publish?	Were definitions generated?
connectors/source-kustomer-singer	❌	❌

if you have connectors that successfully published but failed definition generation, follow step 4 here ▶️

[sajarin]

/publish connector=connectors/source-kustomer-singer

🕑 Publishing the following connectors:
connectors/source-kustomer-singer
https://github.com/airbytehq/airbyte/actions/runs/2797638092

Connector	Did it publish?	Were definitions generated?
connectors/source-kustomer-singer	❌	❌

if you have connectors that successfully published but failed definition generation, follow step 4 here ▶️

[marcosmarxm]

Sorry the delay here @ivansabik we're having problems publishing Singer based connectors. As soon we solve the problem 'll return here

[ivansabik]

No problem, let me know if you need me to rebase or change anything :)

[marcosmarxm]

@ivansabik I'm closing this in favor of in favor of #16480 which implements Kustomer using native Airbyte CDK.

[ivansabik]

Great news! I found other issues with the Kustomer tap and looks like it has not been maintained in a while