-
Notifications
You must be signed in to change notification settings - Fork 4.4k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Docs: streams table #24326
Merged
Merged
Docs: streams table #24326
Changes from all commits
Commits
Show all changes
13 commits
Select commit
Hold shift + click to select a range
665d7d1
Added updated text
sophia-wiley 40748de
edits
sophia-wiley 6213d82
edits to text
sophia-wiley 59f2195
removed header
sophia-wiley e16e04c
edited links
sophia-wiley eccb4da
edited some text
sophia-wiley 13eb34d
added new gifs to doc
sophia-wiley d041518
typo fixes
sophia-wiley eb693dd
Update docs/cloud/managing-airbyte-cloud/edit-stream-configuration.md
sophia-wiley 58eea71
Update docs/cloud/managing-airbyte-cloud/edit-stream-configuration.md
sophia-wiley 3d6d230
Update edit-stream-configuration.md
sophia-wiley 9677e80
Update docs/cloud/managing-airbyte-cloud/edit-stream-configuration.md
sophia-wiley 3885330
Update edit-stream-configuration.md
sophia-wiley File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
150 changes: 109 additions & 41 deletions
150
docs/cloud/managing-airbyte-cloud/edit-stream-configuration.md
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change | ||||
---|---|---|---|---|---|---|
@@ -1,94 +1,162 @@ | ||||||
# Edit stream configuration | ||||||
# Manage syncs | ||||||
|
||||||
By editing stream configurations, you can customize how your data syncs to the destination. This allows you to manage aspects of the sync, such as replication frequency, destination namespace, and ensuring your data is written to the correct location. | ||||||
After you have created a connection, you can change how your data syncs to the destination by modifying the [configuration settings](#configure-connection-settings) and the [stream settings](#modify-streams-in-your-connection). | ||||||
|
||||||
1. On the [Airbyte Cloud](http://cloud.airbyte.com) dashboard, click **Connections** and then click the connection you want to change. | ||||||
## Configure connection settings | ||||||
|
||||||
Configuring the connection settings allows you to manage various aspects of the sync, such as how often data syncs and where data is written. | ||||||
|
||||||
To configure these settings: | ||||||
|
||||||
1. On the [Airbyte Cloud](http://cloud.airbyte.com) dashboard, click **Connections** and then click the connection you want to change. | ||||||
|
||||||
2. Click the **Replication** tab. | ||||||
|
||||||
The **Transfer** and **Streams** settings include the following parameters: | ||||||
3. Click the **Configuration** dropdown. | ||||||
|
||||||
You can configure the following settings: | ||||||
|
||||||
| Parameter | Description | | ||||||
:::note | ||||||
|
||||||
These settings apply to all streams in the connection. | ||||||
|
||||||
::: | ||||||
|
||||||
| Setting | Description | | ||||||
|--------------------------------------|-------------------------------------------------------------------------------------| | ||||||
| Replication frequency | How often the data syncs | | ||||||
| Destination namespace | Where the replicated data is written | | ||||||
| Destination stream prefix | How you identify streams from different connectors | | ||||||
| [Non-breaking schema updates](https://docs.airbyte.com/cloud/managing-airbyte-cloud/manage-schema-changes/#review-non-breaking-schema-changes) detected | How Airbyte handles syncs when it detects non-breaking schema changes in the source | | ||||||
| Destination Namespace | Where the replicated data is written | | ||||||
| Destination Stream Prefix | Helps you identify streams from different connectors | | ||||||
|
||||||
:::note | ||||||
|
||||||
These parameters apply to all streams in the connection. | ||||||
To use [cron scheduling](http://www.quartz-scheduler.org/documentation/quartz-2.3.0/tutorials/crontrigger.html): | ||||||
|
||||||
::: | ||||||
|
||||||
If you need to use [cron scheduling](http://www.quartz-scheduler.org/documentation/quartz-2.3.0/tutorials/crontrigger.html): | ||||||
1. In the **Replication Frequency** dropdown, click **Cron**. | ||||||
|
||||||
2. Enter a cron expression and choose a time zone to create a sync schedule. | ||||||
|
||||||
:::note | ||||||
|
||||||
* Only one sync per connection can run at a time. | ||||||
* If cron schedules a sync to run before the last one finishes, the scheduled sync will start after the last sync completes. | ||||||
* Airbyte Cloud does not allow schedules that sync more than once per hour. | ||||||
* If a sync is scheduled to run before the previous sync finishes, the scheduled sync will start after the completion of the previous sync. | ||||||
* Airbyte Cloud does not support schedules that sync more frequently than once per hour. | ||||||
|
||||||
::: | ||||||
|
||||||
In the **Activate the streams you want to sync section**, you can make changes to any stream you choose. | ||||||
## Modify streams in your connection | ||||||
|
||||||
sophia-wiley marked this conversation as resolved.
Show resolved
Hide resolved
|
||||||
To search for a stream: | ||||||
In the **Activate the streams you want to sync** table, you can choose which streams to sync and how they are loaded to the destination. | ||||||
|
||||||
1. Click the **Search stream name** search box. | ||||||
:::note | ||||||
A connection's schema consists of one or many streams. Each stream is most commonly associated with a database table or an API endpoint. Within a stream, there can be one or many fields or columns. | ||||||
::: | ||||||
|
||||||
To modify streams: | ||||||
|
||||||
1. On the [Airbyte Cloud](http://cloud.airbyte.com) dashboard, click **Connections** and then click the connection you want to change. | ||||||
|
||||||
2. Click the **Replication** tab. | ||||||
|
||||||
3. Scroll down to the **Activate the streams you want to sync** table. | ||||||
|
||||||
:::note | ||||||
|
||||||
2. Type the name of the stream you want to find. | ||||||
You can modify a single stream, or you can modify multiple streams at once. | ||||||
|
||||||
3. Streams matching your search are displayed in the list. | ||||||
::: | ||||||
|
||||||
To change individual stream configuration: | ||||||
Modify an individual stream: | ||||||
|
||||||
 | ||||||
 | ||||||
|
||||||
1. In the **Sync** column of the stream, toggle the sync on or off. | ||||||
1. In the **Activate the streams you want to sync** table, toggle **Sync** on or off for your selected stream. | ||||||
|
||||||
2. Click the dropdown arrow in the **Sync mode** column and select the sync mode you want to apply. | ||||||
2. Click the **Sync mode** dropdown and select the sync mode you want to apply. | ||||||
|
||||||
:::note | ||||||
|
||||||
Depending on the sync mode you select, you may need to choose a cursor or primary key. | ||||||
|
||||||
::: | ||||||
|
||||||
3. If there is a dropdown arrow in the **Cursor** or **Primary key** fields, click the dropdown arrow and choose the cursor or primary key. | ||||||
3. Select the **Cursor** or **Primary keys** when they are required by the stream sync mode. | ||||||
|
||||||
To change multiple stream configurations: | ||||||
:::note | ||||||
|
||||||
 | ||||||
Source-defined cursors and primary keys are selected automatically and cannot be changed in the table. | ||||||
|
||||||
1. Click the first checkbox in the table header to select all streams in the connection. | ||||||
|
||||||
2. Deselect the checkboxes of streams you do not want to apply these changes to. | ||||||
::: | ||||||
|
||||||
4. Click on a stream to display the stream details panel. | ||||||
|
||||||
5. Toggle individual fields to include or exclude them in the sync, or use the toggle in the table header to select all fields at once. | ||||||
|
||||||
3. In the highlighted header of the table, toggle the sync on or off. | ||||||
:::note | ||||||
|
||||||
* You can only deselect top-level fields. You cannot deselect nested fields. | ||||||
* The Airbyte platform may read all data from the source (depending on the source), but it will only write data to the destination from fields you selected. Deselecting fields will not prevent the Airbyte platform from reading them. | ||||||
* When you refresh the schema, newly added fields will be selected by default, even if you have previously deselected fields in that stream. | ||||||
|
||||||
4. Click the dropdown arrow in the **Sync mode** column and select the sync mode you want to apply to these streams. | ||||||
::: | ||||||
|
||||||
5. If there is a dropdown arrow in the **Cursor** or **Primary key** fields of the highlighted table header, click the dropdown arrow and choose the cursor or primary key. | ||||||
6. Depending on the sync mode you chose for your connection, you can select the **Cursor** or **Primary keys** for individual fields in this table. | ||||||
|
||||||
6. Click **Apply** to apply these changes to the streams you selected, or click **Cancel** to discard the changes. | ||||||
7. Click the **X** to close the stream details panel. | ||||||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
|
||||||
|
||||||
To save the changes: | ||||||
1. Click **Save changes**, or click **Cancel** to discard the changes. | ||||||
8. Click **Save changes**, or click **Cancel** to discard the changes. | ||||||
|
||||||
2. The **Stream configuration changed** dialog displays. This gives you the option to reset streams when you save the changes. | ||||||
9. The **Stream configuration changed** dialog displays. This gives you the option to reset streams when you save the changes. | ||||||
|
||||||
:::caution | ||||||
|
||||||
Airbyte recommends that you reset streams. A reset will delete data in the destination of the affected streams and then re-sync that data. Skipping a reset is discouraged and might lead to unexpected behavior. | ||||||
|
||||||
::: | ||||||
|
||||||
3. Click **Save connection**, or click **Cancel** to close the dialog. | ||||||
10. Click **Save connection**. | ||||||
|
||||||
Modify multiple streams: | ||||||
|
||||||
 | ||||||
|
||||||
1. In the **Activate the streams you want to sync** table, select the checkboxes of streams that you want to apply changes to. | ||||||
|
||||||
:::note | ||||||
|
||||||
To refresh the source schema: | ||||||
1. Click **Refresh source schema** to fetch the schema of your data source. | ||||||
To select or deselect all streams, click the checkbox in the table header. To deselect an individual stream, deselect its checkbox in the table. | ||||||
|
||||||
::: | ||||||
|
||||||
* In the highlighted footer of the table: | ||||||
|
||||||
1. Toggle **Sync** on or off. | ||||||
|
||||||
2. Click the **Sync mode** dropdown and select the sync mode you want to apply. | ||||||
|
||||||
:::note | ||||||
|
||||||
Depending on the sync mode you select, you may need to choose a cursor or primary key. | ||||||
|
||||||
::: | ||||||
|
||||||
3. Select the **Cursor** and **Primary keys** if there are dropdowns in those fields. | ||||||
|
||||||
:::note | ||||||
|
||||||
Source-defined cursors and primary keys cannot be changed while configuring multiple streams. | ||||||
|
||||||
::: | ||||||
|
||||||
4. Click **Apply** to apply these changes to the streams you selected, or click **Cancel** to discard the changes. | ||||||
|
||||||
2. Click **Save changes**, or click **Cancel** to discard the changes. | ||||||
|
||||||
3. The **Stream configuration changed** dialog displays. This gives you the option to reset streams when you save the changes. | ||||||
|
||||||
:::caution | ||||||
|
||||||
Airbyte recommends that you reset streams. A reset will delete data in the destination of the affected streams and then re-sync that data. Skipping a reset is discouraged and might lead to unexpected behavior. | ||||||
|
||||||
::: | ||||||
|
||||||
2. If the schema has changed, the **Refreshed source schema** dialog displays them. | ||||||
4. Click **Save connection**. |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Not sure if manage syncs is a good name for this. We're not managing the syncs here. What was there before makes more sense to me.