Tutorial and documentation for config-based connectors #15027

girarda · 2022-07-26T00:28:49Z

What

Add a tutorial and some documentation to help developers implement config-based connectors

How

6-step tutorial implementing a source connector for exchange api
High level documentation describing the components that can be used (with some examples)
Everything is bundled in docs/connector-development/config-based/ since I'm not sure we want everyone to start using config-based right away. Happy to move the files to wherever makes the most sense though.

🚨 User Impact 🚨

Are there any breaking changes? What is the end result perceived by the user? If yes, please merge this PR with the 🚨🚨 emoji so changelog authors can further highlight this if needed.

Pre-merge Checklist

Expand the relevant checklist and delete the others.

New Connector

Community member or Airbyter

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

Create a non-forked branch based on this PR and test the below items on it
Build is successful
If new credentials are required for use in CI, add them to GSM. Instructions.
/test connector=connectors/<name> command is passing
New Connector version released on Dockerhub by running the /publish command described here
After the connector is published, connector added to connector index as described here
Seed specs have been re-generated by building the platform and committing the changes to the seed spec files, as described here

Updating a connector

Community member or Airbyter

Grant edit access to maintainers (instructions)
Secrets in the connector's spec are annotated with airbyte_secret
Unit & integration tests added and passing. Community members, please provide proof of success locally e.g: screenshot or copy-paste unit, integration, and acceptance test output. To run acceptance tests for a Python connector, follow instructions in the README. For java connectors run ./gradlew :airbyte-integrations:connectors:<name>:integrationTest.
Code reviews completed
Documentation updated
- Connector's README.md
- Connector's bootstrap.md. See description and examples
- Changelog updated in docs/integrations/<source or destination>/<name>.md including changelog. See changelog example
PR name follows PR naming conventions

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

Create a non-forked branch based on this PR and test the below items on it
Build is successful
If new credentials are required for use in CI, add them to GSM. Instructions.
/test connector=connectors/<name> command is passing
New Connector version released on Dockerhub and connector version bumped by running the /publish command described here

Connector Generator

Issue acceptance criteria met
PR name follows PR naming conventions
If adding a new generator, add it to the list of scaffold modules being tested
The generator test modules (all connectors with -scaffold in their name) have been updated with the latest scaffold by running ./gradlew :airbyte-integrations:connector-templates:generator:testScaffoldTemplates then checking in your changes
Documentation which references the generator is updated as needed

Tests

Unit

Put your unit tests output here.

Integration

Put your integration tests output here.

Acceptance

Put your acceptance tests output here.

girarda · 2022-07-26T00:31:53Z

docs/connector-development/tutorials/cdk-api-source/config-based/1-create-source.md

+Let's start by cloning the Airbyte repository
+
+```
+git clone git@github.com:airbytehq/airbyte.git


the instructions depend on merging #14552 and #14923

girarda · 2022-08-01T23:08:09Z

docs/connector-development/config-based/tutorial/6-testing.md

+python -m pytest integration_tests -p integration_tests.acceptance
+```
+
+1 test should be failing


there are currently 2 tests failing. the second one will be fixed by #15067

I confirm I have the following two failures:

- airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_core.py:183 TestConnection.test_check[inputs1] - airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_incremental.py:207 TestIncremental.test_state_with_abnormally_large

girarda · 2022-08-01T23:50:44Z

docs/connector-development/config-based/connector-definition.md

+1. `streams`: list of streams that are part of the source
+2. `check`: component describing how to check the connection.
+
+The configuration will be validated against this JSON Schema, which defines the set of valid properties.


soon (tm).
we don't have the json schema yet

brianjlai

Added a few suggestions for grammar fixes but looks good to me!

brianjlai · 2022-08-12T19:50:36Z

docs/connector-development/config-based/overview.md

+4. [Data retriever](overview.md#data-retriever): Describes how to retrieve the data from the API
+5. [Cursor field](../cdk-python/incremental-stream.md) (Optional): Field to use used as stream cursor. Can either be a string, or a list of strings if the cursor is a nested field.
+6. [Transformations](./record-selector.md#transformations) (Optional): A set of transformations to be applied on the records read from the source before emitting them to the destination
+7. [Checkpoint interval](https://docs.airbyte.com/understanding-airbyte/airbyte-protocol/#state--checkpointing) (Optional): Defines the interval at which incremental syncs should be checkpointed.


It might be worth providing an example of what the integer indicates. For example, 5 could mean a lot of different things, items, hours, days, etc

agreed. done

docs/connector-development/config-based/overview.md

brianjlai · 2022-08-12T20:33:57Z

docs/connector-development/config-based/pagination.md

+
+This cursor value can be used to request the next page of record.
+
+In this example, the next page of record is defined by setting the `from` request parameter to the id of the last record read:


I think it might be worth adding a sub header to break apart the two approaches to pagination called "Cursor Pagination in Body" and "Cursor Pagination in Headers", just to differentiate them from a reading perspective

docs/connector-development/config-based/record-selector.md

brianjlai · 2022-08-12T20:44:39Z

docs/connector-development/config-based/request-options.md

+
+- stream_slice
+- stream_state
+- next_page_token


I'm trying to think what the current use case for using these in the language is now, that the preferred method is to let other components handle things more declaratively. My worry is that we highlight this instead of the preferred way. But happy to leave it in if it provides needed flexibility.

good idea. let's leave this as an undocumented behavior for now. we can always add it back with some examples if needed

brianjlai · 2022-08-12T21:08:35Z

docs/connector-development/config-based/stream-slicers.md

+
+will create one slice per day for the interval `2021-02-01` - `2021-03-01`.
+
+The `DatetimeStreamSlicer` also supports an optional lookback window, specifying how many days before the start_datetime to read data for.


as per slack discussion as something we should fix, we should also include that in the case of incremental syncs, the lookback window is also applied for the cursor

created an issue #15628

Co-authored-by: Brian Lai <51336873+brianjlai@users.noreply.github.com>

supertopher

looks sane. I would give a heads up to @Amruta-Ranade especially as this will probably induce unexpected behavior for what docs show up for her and her team

the docs have been updated and reviewed since your last pass

* 5-step tutorial * move * tiny bit of editing * Update tutorial * update docs * reset * move files * record selector, request options, and more links * update * update * connector definition * link * links * update example * footnote * typo * document string interpolation * note on string interpolation * update * fix code sample * fix * update sample * fix * use the actual config * Update as per comments * write as yaml * typo * Clarify options overloading * clarify that docker must be running * remove extra footnote * use venv directly * Apply suggestions from code review Co-authored-by: Sherif A. Nada <snadalive@gmail.com> * signup instructions * update * clarify that both dot and bracket notations are interchangeable * Clarify how check works * create spec and config before updating connector definition * clarify what now_local() is * rename to yaml structure * Go through tutorial and update end of section code samples * fix link * update * update code samples * Update code samples * Update to bracket notation * remove superfluous comments * Update docs/connector-development/config-based/tutorial/2-install-dependencies.md Co-authored-by: Augustin <augustin.lafanechere@gmail.com> * Update docs/connector-development/config-based/tutorial/3-connecting-to-the-API-source.md Co-authored-by: Augustin <augustin.lafanechere@gmail.com> * Update docs/connector-development/config-based/tutorial/3-connecting-to-the-API-source.md Co-authored-by: Augustin <augustin.lafanechere@gmail.com> * Update docs/connector-development/config-based/tutorial/3-connecting-to-the-API-source.md Co-authored-by: Augustin <augustin.lafanechere@gmail.com> * Update docs/connector-development/config-based/tutorial/3-connecting-to-the-API-source.md Co-authored-by: Augustin <augustin.lafanechere@gmail.com> * Update docs/connector-development/config-based/tutorial/3-connecting-to-the-API-source.md Co-authored-by: Augustin <augustin.lafanechere@gmail.com> * Update docs/connector-development/config-based/tutorial/4-reading-data.md Co-authored-by: Augustin <augustin.lafanechere@gmail.com> * fix path * update * motivation blurp * warning * warning * fix code block * update code samples * update code sample * update code samples * small updates * update yaml structure * custom class example * language annotations * update warning * Update tutorial to use dpath extractor * Update record selector docs * unit test * link to contributing * tiny update * $ in front of commands * $ in front of commands * More readings * link to existing config-based connectors * index * update * delete broken link * supported features * update * Add some links * Update docs/connector-development/config-based/overview.md Co-authored-by: Brian Lai <51336873+brianjlai@users.noreply.github.com> * Update docs/connector-development/config-based/record-selector.md Co-authored-by: Brian Lai <51336873+brianjlai@users.noreply.github.com> * Update docs/connector-development/config-based/overview.md Co-authored-by: Brian Lai <51336873+brianjlai@users.noreply.github.com> * Update docs/connector-development/config-based/overview.md Co-authored-by: Brian Lai <51336873+brianjlai@users.noreply.github.com> * Update docs/connector-development/config-based/overview.md Co-authored-by: Brian Lai <51336873+brianjlai@users.noreply.github.com> * mention the unit * headers * remove mentions of interpolating on stream slice, etc. * update * exclude config-based docs Co-authored-by: Sherif A. Nada <snadalive@gmail.com> Co-authored-by: Augustin <augustin.lafanechere@gmail.com> Co-authored-by: Brian Lai <51336873+brianjlai@users.noreply.github.com>

5-step tutorial

4855a72

github-actions bot added the area/documentation Improvements or additions to documentation label Jul 26, 2022

girarda commented Jul 26, 2022

View reviewed changes

girarda added 5 commits July 25, 2022 17:59

move

138bd52

tiny bit of editing

637c2a7

Merge branch 'master' into alex/lowcodeTutorial

9fabad8

Update tutorial

ff775e3

update docs

6ebee74

github-actions bot added the CDK Connector Development Kit label Aug 1, 2022

girarda added 3 commits August 1, 2022 12:14

reset

ff2b602

move files

906f915

record selector, request options, and more links

a64c758

girarda commented Aug 1, 2022

View reviewed changes

girarda added 4 commits August 1, 2022 16:14

update

2099b24

update

8bab845

connector definition

a03e6f3

link

d444d74

girarda commented Aug 1, 2022

View reviewed changes

girarda added 9 commits August 1, 2022 17:02

links

71e0a5b

Merge branch 'master' into alex/lowcodeTutorial

a9512ab

update example

7b36ca6

footnote

218bfd9

typo

78599e3

document string interpolation

c9bfb99

note on string interpolation

58567c5

update

76a95ae

Merge branch 'master' into alex/lowcodeTutorial

8feb1a6

girarda changed the title ~~[low-code connectors] tutorial~~ Tutorial and documentation for config-based connectors Aug 2, 2022

girarda requested review from brianjlai and sherifnada August 2, 2022 02:21

girarda added 9 commits August 12, 2022 08:13

$ in front of commands

b2040fc

More readings

db243a8

link to existing config-based connectors

cc0d76c

index

6cbdaa0

update

619bf37

delete broken link

9a4f1c9

supported features

5337868

update

e4919d5

Add some links

e27fade

brianjlai approved these changes Aug 12, 2022

View reviewed changes

girarda and others added 8 commits August 12, 2022 14:47

Update docs/connector-development/config-based/overview.md

048bddb

Co-authored-by: Brian Lai <51336873+brianjlai@users.noreply.github.com>

Update docs/connector-development/config-based/record-selector.md

019aad2

Co-authored-by: Brian Lai <51336873+brianjlai@users.noreply.github.com>

Update docs/connector-development/config-based/overview.md

cc308f2

Co-authored-by: Brian Lai <51336873+brianjlai@users.noreply.github.com>

Update docs/connector-development/config-based/overview.md

e3cffc8

Co-authored-by: Brian Lai <51336873+brianjlai@users.noreply.github.com>

Update docs/connector-development/config-based/overview.md

785a3e4

Co-authored-by: Brian Lai <51336873+brianjlai@users.noreply.github.com>

mention the unit

2db7694

headers

2a7d5fc

remove mentions of interpolating on stream slice, etc.

eba0322

girarda mentioned this pull request Aug 12, 2022

[low-code connectors] lookback window should be applied on the cursor instead of on the start date #15628

Closed

1 task

Merge branch 'master' into alex/lowcodeTutorial

e7f0023

girarda requested a review from alafanechere August 12, 2022 22:01

girarda added 2 commits August 12, 2022 15:09

update

8353121

exclude config-based docs

e6637d3

supertopher approved these changes Aug 12, 2022

View reviewed changes

girarda merged commit 288c3ca into master Aug 12, 2022

girarda deleted the alex/lowcodeTutorial branch August 12, 2022 22:50

octavia-squidington-iii mentioned this pull request Aug 19, 2022

Bump Airbyte version from 0.40.0-alpha to 0.40.1-alpha #15775

Closed

evantahler mentioned this pull request Aug 22, 2022

Bump Airbyte version from 0.40.0-alpha to 0.40.1 #15857

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Tutorial and documentation for config-based connectors #15027

Tutorial and documentation for config-based connectors #15027

girarda commented Jul 26, 2022 •

edited

Loading

girarda Jul 26, 2022

girarda Aug 1, 2022

alafanechere Aug 3, 2022

girarda Aug 1, 2022

brianjlai left a comment

brianjlai Aug 12, 2022

girarda Aug 12, 2022

brianjlai Aug 12, 2022

girarda Aug 12, 2022

brianjlai Aug 12, 2022

girarda Aug 12, 2022

brianjlai Aug 12, 2022

girarda Aug 12, 2022

supertopher left a comment


		This cursor value can be used to request the next page of record.

		In this example, the next page of record is defined by setting the `from` request parameter to the id of the last record read:


		will create one slice per day for the interval `2021-02-01` - `2021-03-01`.

		The `DatetimeStreamSlicer` also supports an optional lookback window, specifying how many days before the start_datetime to read data for.

Tutorial and documentation for config-based connectors #15027

Tutorial and documentation for config-based connectors #15027

Conversation

girarda commented Jul 26, 2022 • edited Loading

What

How

Recommended reading order

🚨 User Impact 🚨

Pre-merge Checklist

Community member or Airbyter

Airbyter

Community member or Airbyter

Airbyter

Tests

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

brianjlai left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

supertopher left a comment

Choose a reason for hiding this comment

girarda commented Jul 26, 2022 •

edited

Loading