implement column filtering in the replication workflow #20369
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
octavia-squidington-iv
added
area/platform
pmossman
reviewed
Dec 12, 2022
There was a problem hiding this comment.
@mfsiega-airbyte high level approach looks solid to me! I left some comments. Let me know when you've added tests and I can do another review with an explicit approval. Also, were you planning on adding an Acceptance Test in this PR as well?
airbyte-commons-worker/src/main/java/io/airbyte/workers/general/DefaultReplicationWorker.java
Outdated
Show resolved
Hide resolved
| @@ -279,18 +287,24 @@ private static Runnable readFromDstRunnable(final AirbyteDestination destination | |||
| @SuppressWarnings("PMD.AvoidInstanceofChecksInCatchClause") | |||
| private static Runnable readFromSrcAndWriteToDstRunnable(final AirbyteSource source, | |||
| final AirbyteDestination destination, | |||
| ConfiguredAirbyteCatalog catalog, | |||
airbyte-commons-worker/src/main/java/io/airbyte/workers/general/DefaultReplicationWorker.java
Outdated
Show resolved
Hide resolved
airbyte-commons-worker/src/main/java/io/airbyte/workers/general/DefaultReplicationWorker.java
Outdated
Show resolved
Hide resolved
airbyte-commons-worker/src/main/java/io/airbyte/workers/general/DefaultReplicationWorker.java
Outdated
Show resolved
Hide resolved
airbyte-commons-worker/src/main/java/io/airbyte/workers/general/DefaultReplicationWorker.java
Show resolved
Hide resolved
airbyte-commons-worker/src/main/java/io/airbyte/workers/general/DefaultReplicationWorker.java
Show resolved
Hide resolved
airbyte-commons/src/main/java/io/airbyte/commons/features/EnvVariableFeatureFlags.java
Outdated
Show resolved
Hide resolved
mfsiega-airbyte
temporarily deployed
to
more-secrets
GitHub Actions
Inactive
mfsiega-airbyte
temporarily deployed
to
more-secrets
GitHub Actions
Inactive
mfsiega-airbyte
temporarily deployed
to
more-secrets
GitHub Actions
Inactive
mfsiega-airbyte
temporarily deployed
to
more-secrets
GitHub Actions
Inactive
mfsiega-airbyte
temporarily deployed
to
more-secrets
GitHub Actions
Inactive
mfsiega-airbyte
temporarily deployed
to
more-secrets
GitHub Actions
Inactive
mfsiega-airbyte
temporarily deployed
to
more-secrets
GitHub Actions
Inactive
mfsiega-airbyte
temporarily deployed
to
more-secrets
GitHub Actions
Inactive
|
@pmossman I think this should be ready for review, marked it as such! The acceptance test passes locally; if I have issues getting it to pass in CI I'd say go ahead without it in this PR (since the feature is disabled by default) and I can work it out after. |
mfsiega-airbyte
commented
Dec 12, 2022
airbyte-commons-worker/src/main/java/io/airbyte/workers/general/DefaultReplicationWorker.java
Show resolved
Hide resolved
pmossman
approved these changes
Dec 12, 2022
airbyte-commons-worker/src/main/java/io/airbyte/workers/general/DefaultReplicationWorker.java
Outdated
Show resolved
Hide resolved
| @@ -334,6 +336,68 @@ void testOnlyStateAndRecordMessagesDeliveredToDestination() throws Exception { | |||
| verify(destination, never()).accept(TRACE_MESSAGE); | |||
| } | |||
|
|
|||
| @Test | |||
benmoriceau
reviewed
Dec 12, 2022
mfsiega-airbyte
temporarily deployed
to
more-secrets
GitHub Actions
Inactive
mfsiega-airbyte
temporarily deployed
to
more-secrets
GitHub Actions
Inactive
mfsiega-airbyte
deleted the
msiega/column-selection-replication-workflow
branch
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What
Implement column selection in the replication workflow.
It is behind a feature flag, default disabled.
How
If field selection is enabled, only retain the fields in the message from the source that are explicitly included in the catalog.
Recommended reading order
airbyte-commons-worker/src/main/java/io/airbyte/workers/general/DefaultReplicationWorker.java