-
Notifications
You must be signed in to change notification settings - Fork 4.4k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Update low-code documentation #15748
Changes from 4 commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -51,12 +51,12 @@ A stream generally corresponds to a resource within the API. They are analogous | |
A stream is defined by: | ||
|
||
1. A name | ||
2. Primary key (Optional): Used to uniquely identify records, enabling deduplication. Can be a string for single primary keys, a list of strings for composite primary keys, or a list of list of strings for composite primary keys consisting of nested fields. | ||
2. Primary key (Optional): Used to uniquely identify records, enabling deduplication. Can be a string for single primary keys, a list of strings for composite primary keys, or a list of list of strings for composite primary keys consisting of nested fields | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. remove dots for consistency |
||
3. [Schema](../cdk-python/schemas.md): Describes the data to sync | ||
4. [Data retriever](overview.md#data-retriever): Describes how to retrieve the data from the API | ||
5. [Cursor field](../cdk-python/incremental-stream.md) (Optional): Field to use used as stream cursor. Can either be a string, or a list of strings if the cursor is a nested field. | ||
5. [Cursor field](../cdk-python/incremental-stream.md) (Optional): Field to use used as stream cursor. Can either be a string, or a list of strings if the cursor is a nested field | ||
6. [Transformations](./record-selector.md#transformations) (Optional): A set of transformations to be applied on the records read from the source before emitting them to the destination | ||
7. [Checkpoint interval](https://docs.airbyte.com/understanding-airbyte/airbyte-protocol/#state--checkpointing) (Optional): Defines the interval, in number of records, at which incremental syncs should be checkpointed. | ||
7. [Checkpoint interval](https://docs.airbyte.com/understanding-airbyte/airbyte-protocol/#state--checkpointing) (Optional): Defines the interval, in number of records, at which incremental syncs should be checkpointed | ||
|
||
More details on streams and sources can be found in the [basic concepts section](../cdk-python/basic-concepts.md). | ||
|
||
|
@@ -87,9 +87,11 @@ The `SimpleRetriever`'s data flow can be described as follows: | |
2. Select the records from the response | ||
3. Repeat for as long as the paginator points to a next page | ||
|
||
More details on the record selector can be found in the [record selector section](record-selector.md) | ||
More details on the stream slicers can be found in the [stream slicers section](stream-slicers.md) | ||
More details on the paginator can be found in the [pagination section](pagination.md) | ||
More details on the record selector can be found in the [record selector section](record-selector.md). | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. add newlines because these all show up as a single line in the docs |
||
|
||
More details on the stream slicers can be found in the [stream slicers section](stream-slicers.md). | ||
|
||
More details on the paginator can be found in the [pagination section](pagination.md). | ||
|
||
## Requester | ||
|
||
|
@@ -104,7 +106,8 @@ There is currently only one implementation, the `HttpRequester`, which is define | |
6. An error handler: Defines how to handle errors | ||
|
||
More details on authentication can be found in the [authentication section](authentication.md). | ||
More details on error handling can be found in the [error handling section](error-handling.md) | ||
|
||
More details on error handling can be found in the [error handling section](error-handling.md). | ||
|
||
## Connection Checker | ||
|
||
|
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -9,7 +9,7 @@ When a stream is read incrementally, a state message will be output by the conne | |
At the beginning of a `read` operation, the `StreamSlicer` will compute the slices to sync given the connection config and the stream's current state, | ||
As the `Retriever` reads data from the `Source`, the `StreamSlicer` keeps track of the `Stream`'s state, which will be emitted after reading each stream slice. | ||
|
||
More information of stream slicing can be found in the [stream-slices section](../cdk-python/stream-slices.md) | ||
More information of stream slicing can be found in the [stream-slices section](../cdk-python/stream-slices.md). | ||
|
||
## Implementations | ||
|
||
|
@@ -56,10 +56,9 @@ If the `cursor_field` is `created`, and the record is `{"id": 1234, "created": " | |
|
||
When reading data from the source, the cursor value will be updated to the max datetime between | ||
|
||
- the last record's cursor field | ||
- the start of the stream slice | ||
- the current cursor value | ||
This ensures that the cursor will be updated even if a stream slice does not contain any data. | ||
- The last record's cursor field | ||
- The start of the stream slice | ||
- The current cursor value. This ensures that the cursor will be updated even if a stream slice does not contain any data | ||
|
||
#### Stream slicer on dates | ||
|
||
|
@@ -164,7 +163,7 @@ retriever: | |
stream_slice_field: "repository" | ||
``` | ||
|
||
[^1] This is a slight oversimplification. See update cursor section for more details on how the cursor is updated | ||
[^1] This is a slight oversimplification. See [update cursor section](#cursor-update) for more details on how the cursor is updated. | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. add a link to the section |
||
|
||
## More readings | ||
|
||
|
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -8,7 +8,7 @@ Throughout this tutorial, we'll walk you through the creation an Airbyte source | |
|
||
We'll build a connector reading data from the Exchange Rates API, but the steps will apply to other HTTP APIs you might be interested in integrating with. | ||
|
||
The API documentations can be found [here](https://exchangeratesapi.io/documentation/). | ||
The API documentations can be found [here](https://apilayer.com/marketplace/exchangerates_data-api). | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. point to their updated api docs |
||
In this tutorial, we will read data from the following endpoints: | ||
|
||
- `Latest Rates Endpoint` | ||
|
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -11,13 +11,13 @@ Over the course of this tutorial, we'll be editing a few files that were generat | |
|
||
We'll also be creating the following files: | ||
|
||
- `source-exchange-rates-tutorial/secrets/config.json`: This is the configuration file we'll be using to test the connector. It's schema should match the schema defined in the spec file. | ||
- `source-exchange-rates-tutorial/secrets/invalid_config.json`: This is an invalid configuration file we'll be using to test the connector. It's schema should match the schema defined in the spec file. | ||
- `source-exchange-rates-tutorial/secrets/config.json`: This is the configuration file we'll be using to test the connector. Its schema should match the schema defined in the spec file. | ||
- `source-exchange-rates-tutorial/secrets/invalid_config.json`: This is an invalid configuration file we'll be using to test the connector. Its schema should match the schema defined in the spec file. | ||
- `source_exchange_rates_tutorial/schemas/rates.json`: This is the [schema definition](../../cdk-python/schemas.md) for the stream we'll implement. | ||
|
||
## Updating the connector spec and config | ||
|
||
Let's populate the specification (`spec.yaml`) the configuration (`secrets/config.json), so the connector can access the access key and base currency. | ||
Let's populate the specification (`spec.yaml`) and the configuration (`secrets/config.json`) so the connector can access the access key and base currency. | ||
|
||
1. We'll add these properties to the connector spec in `source-exchange-rates-tutorial/source_exchange_rates_tutorial/spec.yaml` | ||
|
||
|
@@ -61,9 +61,9 @@ $ echo '{"access_key": "<your_access_key>", "base": "USD"}' > secrets/config.js | |
Next, we'll update the connector definition (`source-exchange-rates-tutorial/source_exchange_rates_tutorial/exchange_rates_tutorial.yaml`). It was generated by the code generation script. | ||
More details on the connector definition file can be found in the [overview](../overview.md) and [connection definition](../yaml-structure.md) sections. | ||
|
||
Let's fill this out these TODOs with the information found in the [Exchange Rates API docs](https://exchangeratesapi.io/documentation/) | ||
Let's fill this out these TODOs with the information found in the [Exchange Rates API docs](https://apilayer.com/marketplace/exchangerates_data-api). | ||
|
||
1. First, let's rename the stream from `customers` to `rates`, and update the primary key to `date` | ||
1. First, let's rename the stream from `customers` to `rates`, and update the primary key to `date`. | ||
|
||
```yaml | ||
streams: | ||
|
@@ -84,7 +84,7 @@ check: | |
Adding the reference in the `check` tells the `check` operation to use that stream to test the connection. | ||
|
||
2. Next we'll set the base url. | ||
According to the API documentation, the base url is `"https://api.exchangeratesapi.io/v1/"`. | ||
According to the API documentation, the base url is `"https://api.apilayer.com"`. | ||
|
||
```yaml | ||
definitions: | ||
|
@@ -141,7 +141,7 @@ definitions: | |
base: "{{ config['base'] }}" | ||
``` | ||
|
||
The full connection definition should now look like | ||
The full connector definition should now look like | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. this is a connector, not a connection |
||
|
||
```yaml | ||
version: "0.1.0" | ||
|
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -135,7 +135,7 @@ In this example, outer.inner.k2 will evaluate to "MyKey is MyValue" | |
## References | ||
|
||
Strings can contain references to previously defined values. | ||
The parser will dereference these values to produce a complete ConnectionDefinition | ||
The parser will dereference these values to produce a complete object definition. | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
|
||
|
||
References can be defined using a "*ref({arg})" string. | ||
|
||
|
@@ -230,7 +230,7 @@ nested.path: "uh oh" | |
value: "uh oh" | ||
``` | ||
|
||
To resolve the ambiguity, we try looking for the reference key at the top level, and then traverse the structs downward | ||
To resolve the ambiguity, we try looking for the reference key at the top-level, and then traverse the structs downward | ||
until we find a key with the given path, or until there is nothing to traverse. | ||
|
||
More details on referencing values can be found [here](https://airbyte-cdk.readthedocs.io/en/latest/api/airbyte_cdk.sources.declarative.parsers.html?highlight=yamlparser#airbyte_cdk.sources.declarative.parsers.yaml_parser.YamlParser). | ||
|
@@ -264,7 +264,7 @@ This means that both these string templates will evaluate to the same string: | |
1. `"{{ options.name }}"` | ||
2. `"{{ options['name'] }}"` | ||
|
||
In additional to passing additional values through the kwargs argument, macros can be called from within the string interpolation. | ||
In additional to passing additional values through the $options argument, macros can be called from within the string interpolation. | ||
For example, | ||
`"{{ max(2, 3) }}" -> 3` | ||
|
||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
wrong link