🎉 New destination: Streamr

[thanhlmm]

What

Add Streamr destination connector

How

Add Streamr destination using faros-airbyte-cdk

Recommended reading order

1. Readme.md
2. resources/spec.json

🚨 User Impact 🚨

No

Pre-merge Checklist

Expand the relevant checklist and delete the others.

New Connector

Community member or Airbyter

• Community member? Grant edit access to maintainers (instructions)
• Secrets in the connector's spec are annotated with airbyte_secret
• Unit & integration tests added and passing. Community members, please provide proof of success locally e.g: screenshot or copy-paste unit, integration, and acceptance test output. To run acceptance tests for a Python connector, follow instructions in the README. For java connectors run ./gradlew :airbyte-integrations:connectors:<name>:integrationTest.
• Code reviews completed
• Documentation updated
  • Connector's README.md
  • Connector's bootstrap.md. See description and examples
  • docs/SUMMARY.md
  • docs/integrations/<source or destination>/<name>.md including changelog. See changelog example
  • docs/integrations/README.md
  • airbyte-integrations/builds.md
• PR name follows PR naming conventions

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

• Create a non-forked branch based on this PR and test the below items on it
• Build is successful
• Credentials added to Github CI. Instructions.
• /test connector=connectors/<name> command is passing.
• New Connector version released on Dockerhub by running the /publish command described here
• After the connector is published, connector added to connector index as described here
• Seed specs have been re-generated by building the platform and committing the changes to the seed spec files, as described here

Updating a connector

Community member or Airbyter

• Grant edit access to maintainers (instructions)
• Secrets in the connector's spec are annotated with airbyte_secret
• Unit & integration tests added and passing. Community members, please provide proof of success locally e.g: screenshot or copy-paste unit, integration, and acceptance test output. To run acceptance tests for a Python connector, follow instructions in the README. For java connectors run ./gradlew :airbyte-integrations:connectors:<name>:integrationTest.
• Code reviews completed
• Documentation updated
  • Connector's README.md
  • Connector's bootstrap.md. See description and examples
  • Changelog updated in docs/integrations/<source or destination>/<name>.md including changelog. See changelog example
• PR name follows PR naming conventions

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

• Create a non-forked branch based on this PR and test the below items on it
• Build is successful
• Credentials added to Github CI. Instructions.
• /test connector=connectors/<name> command is passing.
• New Connector version released on Dockerhub by running the /publish command described here
• After the new connector version is published, connector version bumped in the seed directory as described here
• Seed specs have been re-generated by building the platform and committing the changes to the seed spec files, as described here

Connector Generator

• Issue acceptance criteria met
• PR name follows PR naming conventions
• If adding a new generator, add it to the list of scaffold modules being tested
• The generator test modules (all connectors with -scaffold in their name) have been updated with the latest scaffold by running ./gradlew :airbyte-integrations:connector-templates:generator:testScaffoldTemplates then checking in your changes
• Documentation which references the generator is updated as needed.

[CLAassistant]

All committers have signed the CLA.

[harshithmullapudi]

@thanhlmm thanks for the contribution

[thanhlmm]

@harshithmullapudi

You're welcome. Please help me review and point out any remaining work so I can make an update.
Thanks

[marcosmarxm]

Hello @thanhlmm this is the first time for a contribution adding a connector using the TypeScript Faros CDK. I'm talking to the team to see the best way to handle, and soon as I got any update I'll let you know.

[thanhlmm]

Hi @marcosmarxm
No worries, take your time!

[harshithmullapudi]

Hey @thanhlmm I will review this PR by EOW. Sorry for the delay

[thanhlmm]

Hi @harshithmullapudi, any update on this?

[fonty1]

@harshithmullapudi - it would be great to have this merged soon so we can give this integration some attention on our socials :)

[harshithmullapudi]

Hey, @fonty1 sorry for the delay. We are just trying to figure out the best way to merge this in our code base. As this will be our first cross-language connector. Will talk to the team and see what we can do.

[fonty1]

Hey, @fonty1 sorry for the delay. We are just trying to figure out the best way to merge this in our code base. As this will be our first cross-language connector. Will talk to the team and see what we can do.

Ah, interesting!

[harshithmullapudi]

Also happy to discuss if you have suggestions for example

1. Ideally we don't want all these config files (eslint.js, prettier) for every source. It would be better if we can common files across all sources
2. Also to figure out how to run tests. Before that what is the test package we want to use and how can we integrate this with our source approval flow.

Otherwise I had a basic look at the PR and looked promising.

[thanhlmm]

Also happy to discuss if you have suggestions for example

1. Ideally we don't want all these config files (eslint.js, prettier) for every source. It would be better if we can common files across all sources
2. Also to figure out how to run tests. Before that what is the test package we want to use and how can we integrate this with our source approval flow.

Otherwise I had a basic look at the PR and looked promising.

1. I think we can do it with some effort. We need to move these config files to the upper level and have a config file to track only Javascript/Typescript integration only (We don't want eslint or prettier mess others integration)
2. To be honest, I don't see much difference in testing libraries in Javascript. So I think jest might be good, it is top library in Javascript world.

[marcosmarxm]

Hello @thanhlmm, sorry for the long delay to return to you. I talked to Sherif about adding connectors using externals CDK`s. At the moment looks is better to add your contribution to the TypeScript CDK repo or you keep in your repo. In TypeScript CDK repo will be much more easy to create Action and Workflows to apply tests.

You can publish your connector in Airbyte Connector list. See example from Jenkins PR from Faros.ai #7837.

Let me know what do you think about it.

[thanhlmm]

Hey @marcosmarxm do you mean that I should keep the source code in my own repo for now, and wait #7837 to be merged?

[harshithmullapudi]

Hey @thanhlmm his point is to

1. You can have this as a source in this repo https://github.com/faros-ai/airbyte-connectors or have it in your repo only and leave it for now until we figure out the right way to do this.
2. You can in the meantime use Add Jenkins source from Faros AI to connector catalog #7837 to publish the connector.

[thanhlmm]

@harshithmullapudi Thanks

Then I will keep the source in my repo and also add it to connector catalog