Add Airbyte Protocol V1 support. #20036
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
gosusnp
requested review from
edgao,
benmoriceau,
alovew,
colesnodgrass and
jdpgrailsdev
jdpgrailsdev
reviewed
Dec 2, 2022
|
|
||
| @Override | ||
| public Version getCurrentVersion() { | ||
| return new Version("1.0.0"); |
There was a problem hiding this comment.
Does it make sense to define these versions somewhere in a constants class? Something like PREVIOUS and CURRENT, as well as specific version numbers so that when we want to change, we can just basically update that class (e.g. point CURRENT to a newly added VERSION_X constant). Feels like otherwise we are going to have a few versions of these strings floating around. Not sure exactly how to reduce the duplication, but it feels like centralizing these would help.
There was a problem hiding this comment.
Those values are tied to migration itself. Currently, we can add V2 that would have a previous v1, current v2.
We currently have
V0, V1 in there.
edgao
reviewed
Dec 2, 2022
...protocol/src/main/java/io/airbyte/commons/protocol/migrations/AirbyteMessageMigrationV1.java
Outdated
Show resolved
Hide resolved
benmoriceau
reviewed
Dec 3, 2022
* Updated normalization simple stream processing to handle new datatypes * Updated normalization nested stream processing to handle new datatypes * Updated normalization nested stream processing to handle new datatypes * Updated normalization drop_scd_catalog processing to handle new datatypes * Updated normalization ephemeral test processing to handle new datatypes * fixed more tests for normalization * fixed more tests for normalization * fixed more tests for normalization * fixed more tests for normalization * fixed more issues * fixed more issues (clickhouse) * fixed more issues * fixed more issues * fixed more issues * added binary type processing for some DBs * cleared commented code and moved some hardcodes to processing as macro * fixed codestyle and cleared commented code * minor refactor * minor refactor * minor refactor * fixed bool cast error * fixed dict->str cast error * fixed is_combining_node cast py check * removed commented code * removed commented code * committed autogenerated normalization_test_output files * committed autogenerated normalization_test_output files (new files) * refactored utils.py * Updated utils.py to use Callable functions and get rid of property_type in is_number and is_bool functions * committed autogenerated normalization_test_output files (new files) * fixed typo in TIMESTAMP_WITH_TIMEZONE_TYPE * updated stream_processor to handle string type first as a wider type * fixed arrays normalization by updating is_simple_property method as per new approaches * format Co-authored-by: Edward Gao <edward.gao@airbyte.io>
Affected Connector ReportNOTE
|
| Connector | Version | Changelog | Publish |
|---|---|---|---|
source-alloydb |
1.0.36 |
✅ | ✅ |
source-alloydb-strict-encrypt |
1.0.36 |
🔵 (ignored) |
🔵 (ignored) |
source-bigquery |
0.2.3 |
✅ | ✅ |
source-clickhouse |
0.1.15 |
✅ | ✅ |
source-clickhouse-strict-encrypt |
0.1.15 |
🔵 (ignored) |
🔵 (ignored) |
source-cockroachdb |
0.1.19 |
✅ | ✅ |
source-cockroachdb-strict-encrypt |
0.1.19 |
🔵 (ignored) |
🔵 (ignored) |
source-db2 |
0.1.17 |
✅ | ✅ |
source-db2-strict-encrypt |
0.1.17 |
🔵 (ignored) |
🔵 (ignored) |
source-dynamodb |
0.1.0 |
✅ | ✅ |
source-jdbc |
0.3.5 |
🔵 (ignored) |
🔵 (ignored) |
source-mongodb-strict-encrypt |
0.1.19 |
🔵 (ignored) |
🔵 (ignored) |
source-mongodb-v2 |
0.1.19 |
✅ | ✅ |
source-mssql |
0.4.28 |
✅ | ✅ |
source-mssql-strict-encrypt |
0.4.28 |
🔵 (ignored) |
🔵 (ignored) |
source-mysql |
1.0.21 |
✅ | ✅ |
source-mysql-strict-encrypt |
1.0.21 |
🔵 (ignored) |
🔵 (ignored) |
source-oracle |
0.3.22 |
✅ | ✅ |
source-oracle-strict-encrypt |
0.3.22 |
🔵 (ignored) |
🔵 (ignored) |
source-postgres |
1.0.42 |
✅ | ✅ |
source-postgres-strict-encrypt |
1.0.42 |
🔵 (ignored) |
🔵 (ignored) |
source-redshift |
0.3.16 |
✅ | ✅ |
source-scaffold-java-jdbc |
0.1.0 |
🔵 (ignored) |
🔵 (ignored) |
source-snowflake |
0.1.29 |
✅ | ✅ |
source-tidb |
0.2.2 |
✅ | ✅ |
- See "Actionable Items" below for how to resolve warnings and errors.
❌ Destinations (16)
| Connector | Version | Changelog | Publish |
|---|---|---|---|
destination-bigquery |
1.2.13 |
✅ | ✅ |
destination-bigquery-denormalized |
1.2.12 |
✅ | ❌ (diff seed version) |
destination-clickhouse |
0.2.2 |
❌ (changelog missing) |
✅ |
destination-clickhouse-strict-encrypt |
0.2.2 |
🔵 (ignored) |
🔵 (ignored) |
destination-jdbc |
0.3.14 |
🔵 (ignored) |
🔵 (ignored) |
destination-mssql |
0.1.22 |
✅ | ✅ |
destination-mssql-strict-encrypt |
0.1.22 |
🔵 (ignored) |
🔵 (ignored) |
destination-mysql |
0.1.20 |
✅ | ✅ |
destination-mysql-strict-encrypt |
❌ 0.1.21(mismatch: 0.1.20) |
🔵 (ignored) |
🔵 (ignored) |
destination-oracle |
0.1.19 |
✅ | ✅ |
destination-oracle-strict-encrypt |
0.1.19 |
🔵 (ignored) |
🔵 (ignored) |
destination-postgres |
0.3.26 |
✅ | ✅ |
destination-postgres-strict-encrypt |
0.3.26 |
🔵 (ignored) |
🔵 (ignored) |
destination-redshift |
0.3.56 |
✅ | ✅ |
destination-snowflake |
0.4.46 |
✅ | ✅ |
destination-tidb |
0.1.0 |
✅ | ✅ |
- See "Actionable Items" below for how to resolve warnings and errors.
👀 Other Modules (1)
- base-normalization
Actionable Items
(click to expand)
| Category | Status | Actionable Item |
|---|---|---|
| Version | ❌ mismatch |
The version of the connector is different from its normal variant. Please bump the version of the connector. |
| ⚠ doc not found |
The connector does not seem to have a documentation file. This can be normal (e.g. basic connector like source-jdbc is not published or documented). Please double-check to make sure that it is not a bug. |
|
| Changelog | ⚠ doc not found |
The connector does not seem to have a documentation file. This can be normal (e.g. basic connector like source-jdbc is not published or documented). Please double-check to make sure that it is not a bug. |
| ❌ changelog missing |
There is no chnagelog for the current version of the connector. If you are the author of the current version, please add a changelog. | |
| Publish | ⚠ not in seed |
The connector is not in the seed file (e.g. source_definitions.yaml), so its publication status cannot be checked. This can be normal (e.g. some connectors are cloud-specific, and only listed in the cloud seed file). Please double-check to make sure that it is not a bug. |
| ❌ diff seed version |
The connector exists in the seed file, but the latest version is not listed there. This usually means that the latest version is not published. Please use the /publish command to publish the latest version. |
* On the fly catalog migration for normalization activity * On the fly catalog migration for job persistence * On the fly migration for standard sync persistence * On the fly migration for airbyte catalogs * Refactor code to share JsonSchema traversal * Add V0 Data type search function * PMD and Format * Fix getOrInsertActorCatalog and ConfigRepositoryE2E tests * Null-proofing CatalogMigrationV1Helper * More null checks * Fix test * Format * Add data type v1 support to the FE * Changes AC test check to check exited ps (#21672) some docker compose changes no longer show exited processes. this broke out test this change should fix master tested in a runner that failed * Move wellknown types mapping to the utility function * use protocolv1 normalization --------- Co-authored-by: Topher Lubaway <asimplechris@gmail.com> Co-authored-by: Edward Gao <edward.gao@airbyte.io>
* Add normalization min version check * Add visible for testing
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What
Enable Airbyte Protocol V1 in the platform.
Closes #16814
How
This leverages the fact that v1 doesn't change the structure of v0. However, we won't be fully using v1 until #19240 is merged.
In order to properly use the v1 objects, we need to have a few tasks to decouple Java connectors from the default version of the protocol objects.
Recommended reading order
Any
🚨 User Impact 🚨
Following this change, as expected but worth highlighting, we will be converting all messages v0->v1->v0 for all connectors until they have been migrated to v1.