Change State using the UI

[Fredehagelund92]

Tell us about the problem you're trying to solve

As a Data Engineer i am used to having control on all parts of the extract/ingest process. However in Airbyte i am missing a feature to manipulate or manual set the state. In some cases you might be working with legacy systems that produce data in the wrong order, which might invalidate the state if the sync happen during this phase.

Another case is that maybe you do not want to extract all the data from the beginning. Maybe the source has so much data that you only want the data from current date and then incremental from there. An argument could be to just create a configcalled start_date and it is then handled by the source, but forces the creator of a source to think about every specific use-case.

Describe the solution you’d like

I'd like to be able to change the state using the UI. From a UI perspective it could be added to the Connection settings page. In the source schemas table we could add another column, just like Sync settings that contain the current state of that schema.

Describe the alternative you’ve considered or used

Export the airbyte configuration file and make your changes to airbyte_db/JOBS.yaml. Then upload the airbyte configuration using the UI. Not the cleanest, but it works.

Additional context

┆Issue is synchronized with this Asana task by Unito

[michel-tricot]

That's a great idea. I wonder if we could display this information in the connections settings page.

[sherifnada]

for context the answer I gave Fred in slack:

1. export your airbyte configuration from the admin UI
2. turn off airbyte docker-compose down
3. untar the downloaded archive
4. inside airbyte_db/JOBS.yaml find the latest successful job run for the connection whose state you want to change. it’s important that this is the latest, because state is considered to be whatever was output by the last successful sync job.
   you should see a state object like so

config:
    sync:
      state:
        state:

update the values in that object to match what you want

6. tar the entire directory
7. turn on airbyte webapp and webserver docker-compose up -d db server webapp
8. go to the admin Ui and import the configuration you just edited
9. run docker-compose down then docker-compose up to run all of airbyte

Not endorsing it, just putting it here in case someone sees this issue before we get around to doing it.

[phamduyly]

Hi all, have anyone tried this.
I have one concern, how do you define successful job? for example, if I am syncing a 80 mil records, the min incremental value is 1970-01-01, but I only want to extract from 2018-01-01. Then how can I do it?
Thank you.

[marcosmarxm]

@phamduyly did you try doing this? If you need some help let me know

[Phlair]

just adding a relevant slack thread to this issue for reference, with slightly revised steps based on Sherif's comment (tested on 0.29.17-alpha)

[miguel-fresh]

Hi @sherifnada , I'm trying to do what You've explained

for context the answer I gave Fred in slack:

1. export your airbyte configuration from the admin UI
2. turn off airbyte docker-compose down
3. untar the downloaded archive
4. inside airbyte_db/JOBS.yaml find the latest successful job run for the connection whose state you want to change. it’s important that this is the latest, because state is considered to be whatever was output by the last successful sync job.
   you should see a state object like so

config:
    sync:
      state:
        state:

update the values in that object to match what you want

6. tar the entire directory
7. turn on airbyte webapp and webserver docker-compose up -d db server webapp
8. go to the admin Ui and import the configuration you just edited
9. run docker-compose down then docker-compose up to run all of airbyte

Not endorsing it, just putting it here in case someone sees this issue before we get around to doing it.

But when I extract files from Airbyte's .gz exported config I get a single file, not a directory:

Has something changed since last year or am I doing something wrong? thanks in advance!

[evantahler]

From backlog grooming: Timebox this to a few days. Make CRUD apis to view and set state, show in the I with lots of danger warnings.

Requirements:

• read only view of the connection state in the UI

Options, if there's time:

1. Allow writing to the state
2. Don't allow editing when a sync is running.
3. hide on cloud
4. validate the state

[evantahler]

Working on adding STATE to the UI in #13394, but we won't allow editing quite yet

[evantahler]

cc @erohmensing - I think our 'copy state' feature will address this

[nataliekwong]

Hi there, Airbyte now allows editing of the connection state in the UI! I will close the issue now that this is resolved. Feel free to comment if you feel it should be re-opened.