Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Added auto-detect schema info to docs #21501

Merged
merged 5 commits into from
Jan 25, 2023
Merged

Added auto-detect schema info to docs #21501

Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
dae18fd
added auto-detect schema info to docs
sophia-wiley Jan 17, 2023
463e12a
small edits
sophia-wiley Jan 18, 2023
6624ab8
added info
sophia-wiley Jan 23, 2023
c4a7703
edits
sophia-wiley Jan 24, 2023
8abf875
edits
sophia-wiley Jan 25, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
88 changes: 80 additions & 8 deletions docs/cloud/managing-airbyte-cloud.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -171,6 +171,7 @@ To view the sync summary:
3. To view the full sync log, click the sync summary dropdown.

### Sync summary

| Data | Description |
|--------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------|
| x GB (also measured in KB, MB) | Amount of data moved during the sync. If basic normalization is on, the amount of data would not change since normalization occurs in the destination. |
Expand All @@ -186,17 +187,18 @@ In a successful sync, the number of emitted records and committed records should

## Edit stream configuration

1. On the [Airbyte Cloud](http://cloud.airbyte.io) dashboard, click **Connections** and then click a connection in the list you want to change.
1. On the [Airbyte Cloud](http://cloud.airbyte.io) dashboard, click **Connections** and then click the connection you want to change.

2. Click the **Replication** tab.

The **Transfer** and **Streams** settings include the following parameters:

| Parameter | Description |
|---------------------------|-------------------------------------------------------------------------------------------------------------------------------------------|
| Replication Frequency | How often the data syncs |
| [Destination Namespace](https://docs.airbyte.com/understanding-airbyte/namespaces/) | Where the replicated data is written |
| Destination Stream Prefix | Helps you identify streams from different connectors |
| Parameter | Description |
|--------------------------------------|-------------------------------------------------------------------------------------|
| Replication frequency | How often the data syncs |
| [Non-breaking schema updates](#review-non-breaking-schema-changes) detected | How Airbyte handles syncs when it detects non-breaking schema changes in the source |
| Destination Namespace | Where the replicated data is written |
| Destination Stream Prefix | Helps you identify streams from different connectors |

:::note

Expand Down Expand Up @@ -274,9 +276,79 @@ Airbyte recommends that you reset streams. A reset will delete data in the desti
To refresh the source schema:
1. Click **Refresh source schema** to fetch the schema of your data source.

2. If there are changes to the schema, the **Refreshed source schema** dialog displays them.
2. If the schema has changed, the **Refreshed source schema** dialog displays them.

## Manage schema changes

Once every 24 hours, Airbyte checks for changes in your source schema and allows you to review the changes and fix breaking changes.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@Amruta-Ranade what do you think about grouping this information in a note here?

:::note

Schema changes are flagged in your connection but are not propagated to your destination.

:::

### Review non-breaking schema changes

To review non-breaking schema changes:
1. On the [Airbyte Cloud](http://cloud.airbyte.com/) dashboard, click **Connections** and select the connection with non-breaking changes (indicated by a **yellow exclamation mark** icon).

2. Click **Review changes**.

3. The **Refreshed source schema** dialog displays the changes.

4. Review the changes and click **OK** to close the dialog.

5. Scroll to the bottom of the page and click **Save changes**.

:::note

By default, Airbyte ignores non-breaking changes and continues syncing. You can configure how Airbyte handles syncs when it detects non-breaking changes by [editing the stream configuration](#edit-stream-configuration).

:::

### Fix breaking schema changes

:::note

Breaking changes can only occur in the **Cursor** or **Primary key** fields.

:::

To review and fix breaking schema changes:
1. On the [Airbyte Cloud](http://cloud.airbyte.com/) dashboard, click **Connections** and select the connection with breaking changes (indicated by a **red exclamation mark** icon).

2. Click **Review changes**.

3. The **Refreshed source schema** dialog displays the changes.

4. Review the changes and click **OK** to close the dialog.

5. In the streams table, the stream with a breaking change is highlighted.

6. Fix the breaking change by selecting a new **Cursor** or **Primary key**.

7. Scroll to the bottom of the page and click **Save changes**.

:::note

If a connection’s source schema has breaking changes, it will stop syncing. You must review and fix the changes before editing the connection or resuming syncs.

:::

### Enable schema update notifications

To get notified when your source schema changes:
1. Make sure you have [webhook notifications](https://docs.airbyte.com/cloud/managing-airbyte-cloud#manage-airbyte-cloud-notifications) set up.

2. On the [Airbyte Cloud](http://cloud.airbyte.com/) dashboard, click **Connections** and select the connection you want to receive notifications for.

3. Click the **Settings** tab on the Connection page.

4. Toggle **Schema update notifications**.

## Display Connection State

**Connection State** provides additional information about incremental syncs. It includes the most recent values for the global or stream-level cursors, which can aid in debugging or determining which data will be included in the next syncs.

To display **Connection State**:
Expand All @@ -291,7 +363,7 @@ To display **Connection State**:

5. Click the **Settings** tab on the Connection page.

The **Connection State** displays.
The **Connection State** displays.

## Choose the data residency for a connection
You can choose the data residency for your connection in the connection settings. You can also choose data residency when creating a [new connection](https://docs.airbyte.com/cloud/getting-started-with-airbyte-cloud#set-up-a-connection), or you can set the [default data residency](#choose-your-default-data-residency) for your workspace.
Expand Down