Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

📝 add instructions for soft reset #35335

Merged
merged 2 commits into from
Feb 15, 2024
Merged
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 15 additions & 3 deletions docs/using-airbyte/core-concepts/typing-deduping.md
Original file line number Diff line number Diff line change
Expand Up @@ -105,9 +105,9 @@ You also now see the following changes in Airbyte-provided columns:

## On final table re-creation

From time to time, Airbyte will drop and re-create the final table produced by a sync. This is done
as transitionally as possible, and should be invisible to most observers. This is done for a number
of reasons, including:
From time to time, Airbyte will drop and re-create the final table produced by a sync (sometimes
called a "soft reset"). This is done as transactionally as possible, and should be invisible to most
observers. This is done for a number of reasons, including:

- **Schema Migrations** - Many destinations lack the ability to control column order, or cannot
alter one data type to another. Re-creating the table allows Airbyte to strictly control the
Expand All @@ -123,6 +123,18 @@ table outside of Airbyte could be lost during a sync. Many destinations provide
wildcards to grant permissions to tables, which are better suited for this ELT process. We do not
recommend altering the final tables (e.g. adding constraints) as it may cause issues with the sync.

### Manually triggering a final table re-creation

In some cases, you need to manually run a soft reset - for example, if you accidentally delete some
records from the final table and want to repopulate them from the raw data. This can be done by:
1. Dropping the final table entirely (`DROP TABLE <your_final_table>`)
1. Unsetting the raw table's `_airbyte_loaded_at` column
(`UPDATE airbyte_internal.<your_raw_table> SET _airbyte_loaded_at = NULL`)
1. If you are using a nonstandard raw table schema, replace `airbyte_internal` with that schema.
1. And then running a sync.

After the sync completes, your final table will be restored to its correct state.

## Loading Data Incrementally to Final Tables

:::note
Expand Down
Loading