Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Data types update: Implement protocol message migrations #19240

Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
87 commits
Select commit Hold shift + click to select a range
e2146b9
Extract MigrationContainer from AirbyteMessageMigrator
gosusnp Dec 20, 2022
caa1692
Add ConfiguredAirbyteCatalogMigrations
gosusnp Dec 20, 2022
a374a62
Add ConfiguredAirbyteCatalog to AirbyteMessageMigrations
gosusnp Dec 20, 2022
7adbe36
Enable ConfiguredAirbyteCatalog migration
gosusnp Dec 21, 2022
7746f87
set up scaffolding
edgao Nov 8, 2022
06b144a
[wip] more scaffolding, basic unit test
edgao Nov 8, 2022
66c1830
minimal green code
edgao Nov 8, 2022
b9275bd
[wip] add failing test for other primitive types
edgao Nov 8, 2022
a333017
correct version number
edgao Nov 8, 2022
ecbcda9
handle basic primitive type decls
edgao Nov 9, 2022
683e3bf
add implicit cases
edgao Nov 9, 2022
9333641
add recursive schema
edgao Nov 9, 2022
4d71da8
formatting
edgao Nov 9, 2022
e0b95a9
comment
edgao Nov 9, 2022
8ae40ad
support not
edgao Nov 9, 2022
8079601
fix indentation
edgao Nov 9, 2022
8ee6c74
handle all nested schema cases
edgao Nov 9, 2022
916349b
handle boolean schemas
edgao Nov 9, 2022
43fd5df
verify empty schema handling
edgao Nov 9, 2022
82f7a11
cleanup
edgao Nov 9, 2022
8301826
extract map
edgao Nov 9, 2022
66b5141
code organization
edgao Nov 9, 2022
3bd864d
extract method
edgao Nov 9, 2022
0378326
reformat
edgao Nov 9, 2022
d691fe4
[wip] more tests, minor fix type array handling
edgao Nov 9, 2022
06c4ac8
corrected test
edgao Nov 9, 2022
e584a61
cleanup
edgao Nov 9, 2022
3ea8135
reformat
edgao Nov 9, 2022
4c3d960
switch to v1
edgao Nov 10, 2022
a80aa76
add support for multityped fields
edgao Nov 10, 2022
4f242c2
missed test case
edgao Nov 10, 2022
bb5afd5
nested test class
edgao Nov 10, 2022
137ed33
basic record upgrade
edgao Nov 10, 2022
238098b
implement record upgrades
edgao Nov 10, 2022
98cdd89
slight refactor
edgao Nov 10, 2022
2609d88
comments+clarificationso
edgao Nov 11, 2022
c008185
extract constants
edgao Nov 15, 2022
a6c9727
(partly) correct model classes
edgao Nov 16, 2022
32d4c56
add de/ser
edgao Nov 16, 2022
9f868e0
formatting
edgao Nov 16, 2022
ceb4175
extract constants
edgao Nov 16, 2022
71e0eed
fix json reference
edgao Nov 19, 2022
a8e5cce
update docs
edgao Nov 19, 2022
8c79cbd
switch to v1 models
edgao Nov 30, 2022
4475794
fix compile+test
edgao Dec 17, 2022
8e896e5
add base64 handling
edgao Dec 17, 2022
3844a9b
use vnull
edgao Dec 20, 2022
13afb38
Data types update: Implement protocol message downgrade path (#19909)
edgao Dec 21, 2022
5fb55af
move migrations into v1 package
edgao Dec 21, 2022
ed8874a
extract schema mutation code
edgao Dec 21, 2022
9f78cb5
comment
edgao Dec 21, 2022
d8f2465
extract schema migration to new class
edgao Dec 21, 2022
81587a2
extract record downgrade logic for future use
edgao Dec 22, 2022
0c64b7f
format
edgao Dec 22, 2022
b63cce9
fix build after rebase
edgao Dec 23, 2022
433c536
rename private method for consistency
edgao Dec 23, 2022
9478e50
also implement configuredcatalog migrations >.>
edgao Dec 23, 2022
e3a049c
quick and dirty tests
edgao Dec 23, 2022
282f9a1
slight cleanup
edgao Dec 23, 2022
862fa44
fix tests
edgao Dec 23, 2022
eb7b77c
pmd
edgao Dec 23, 2022
60352b7
pmd test
edgao Dec 23, 2022
4ec3ccb
null check on message objects
edgao Dec 23, 2022
7a9ff52
maybe fix acceptance tests?
edgao Dec 23, 2022
a5d49bd
fix name
edgao Jan 2, 2023
7fe61f9
extract constants
edgao Jan 2, 2023
2c3cbbb
more fixes
edgao Jan 4, 2023
774d31c
tmp
edgao Jan 5, 2023
b0270f8
meh
edgao Jan 5, 2023
1300082
fix cdc acc tests
edgao Jan 5, 2023
7f393c7
revert to master source-postgres
edgao Jan 6, 2023
7c5a6dc
remove log messages
edgao Jan 6, 2023
dbad5bb
Merge branch 'gosusnp/platform-use-protocol-v1-the-quick-way' into go…
edgao Jan 6, 2023
f66be38
Merge branch 'gosusnp/20695-update-airbyte-protocol-migration' into l…
edgao Jan 6, 2023
c2a8886
revert other misc hacks
edgao Jan 6, 2023
99d5e64
integers are valid cursors
edgao Jan 6, 2023
7b12861
remove unrelated change
edgao Jan 6, 2023
de287e2
fix build
edgao Jan 6, 2023
5342166
fix build more?
edgao Jan 6, 2023
f703609
[MUST REVERT] use dev normalization
edgao Jan 9, 2023
a7609ca
capture kube logs
edgao Jan 9, 2023
1c91ef7
also here?
edgao Jan 10, 2023
d3d34ea
no debug logs?
edgao Jan 10, 2023
6764a79
Merge branch 'gosusnp/platform-use-protocol-v1-the-quick-way' into ed…
edgao Jan 10, 2023
9c2cf19
delete dup from merging
edgao Jan 10, 2023
f968c6a
add final everywhere
edgao Jan 11, 2023
58f9cd5
revert test changes
edgao Jan 11, 2023
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
24 changes: 12 additions & 12 deletions .github/workflows/gradle.yml
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@ on:
description: "Enable or disable tmate session for debug during helm ac tests"
type: choice
default: 'false'
options:
options:
- 'true'
- 'false'
required: false
Expand Down Expand Up @@ -305,7 +305,7 @@ jobs:
if: always()
run: |
gcs_bucket_name="dev-ab-ci-run-results"
filename=$(echo "${{ fromJSON( steps.connectors-test-results.outputs.json ).check_url }}" | sed 's@.*/@@')
filename=$(echo "${{ fromJSON( steps.connectors-test-results.outputs.json ).check_url }}" | sed 's@.*/@@')
echo "$filename"
gsutil -h "Cache-Control:public" cp connectors_base_results.jsonl "gs://$gcs_bucket_name/oss/$filename.jsonl"

Expand Down Expand Up @@ -699,7 +699,7 @@ jobs:
if: always()
run: |
gcs_bucket_name="dev-ab-ci-run-results"
filename=$(echo "${{ fromJSON( steps.platform-results.outputs.json ).check_url }}" | sed 's@.*/@@')
filename=$(echo "${{ fromJSON( steps.platform-results.outputs.json ).check_url }}" | sed 's@.*/@@')
echo "$filename"
gsutil -h "Cache-Control:public" cp platform_results.jsonl "gs://$gcs_bucket_name/oss/$filename.jsonl"

Expand Down Expand Up @@ -756,7 +756,7 @@ jobs:
github-token: ${{ env.PAT }}
label: ${{ needs.start-platform-build-runner.outputs.label }}
ec2-instance-id: ${{ needs.start-platform-build-runner.outputs.ec2-instance-id }}

## Kube Acceptance Tests
# Docker acceptance tests run as part of the build job.
# In case of self-hosted EC2 errors, remove this block.
Expand Down Expand Up @@ -919,7 +919,7 @@ jobs:
if: always()
run: |
gcs_bucket_name="dev-ab-ci-run-results"
filename=$(echo "${{ fromJSON( steps.kube-results.outputs.json ).check_url }}" | sed 's@.*/@@')
filename=$(echo "${{ fromJSON( steps.kube-results.outputs.json ).check_url }}" | sed 's@.*/@@')
echo "$filename"
gsutil -h "Cache-Control:public" cp kube_results.jsonl "gs://$gcs_bucket_name/oss/$filename.jsonl"

Expand Down Expand Up @@ -1036,7 +1036,7 @@ jobs:
- name: Fix EC-2 Runner
run: |
mkdir -p /actions-runner/_work/airbyte/airbyte && mkdir -p /actions-runner/_work/airbyte/airbyte/.kube

- name: Checkout Airbyte
uses: actions/checkout@v2
with:
Expand All @@ -1059,7 +1059,7 @@ jobs:

- uses: actions/setup-python@v4
with:
python-version: '3.9'
python-version: '3.9'

- uses: actions/setup-java@v1
with:
Expand Down Expand Up @@ -1087,11 +1087,11 @@ jobs:
sudo apt-get -y install tmate
attempt_limit: 3
attempt_delay: 2000 # in ms

- name: Start tmate session in background
if: inputs.debug_mode == 'true'
shell: bash
run: |
run: |
tmate -S /tmp/tmate.sock new-session -d # Launch tmate in a headless mode
tmate -S /tmp/tmate.sock wait tmate-ready # Blocks until the SSH connection is established
tmate -S /tmp/tmate.sock display -p '#{tmate_ssh}' # Prints the SSH connection string
Expand Down Expand Up @@ -1163,7 +1163,7 @@ jobs:
with:
name: Kubernetes Logs
path: /tmp/kubernetes_logs/*

- name: Upload test results to BuildPulse for flaky test detection
if: "!cancelled()" # Run this step even when the tests fail. Skip if the workflow is cancelled.
uses: Workshop64/buildpulse-action@main
Expand All @@ -1173,13 +1173,13 @@ jobs:
path: "/actions-runner/_work/airbyte/airbyte/*"
key: ${{ secrets.BUILDPULSE_ACCESS_KEY_ID }}
secret: ${{ secrets.BUILDPULSE_SECRET_ACCESS_KEY }}

- name: "Display logs of k3s"
if: failure()
shell: bash
run: |
journalctl -xeu k3s.service


# # In case of self-hosted EC2 errors, remove this block.
stop-helm-acceptance-test-runner:
Expand Down
1 change: 1 addition & 0 deletions airbyte-commons-protocol/build.gradle
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@ dependencies {
testImplementation libs.bundles.micronaut.test

implementation project(':airbyte-protocol:protocol-models')
implementation project(':airbyte-json-validation')
}

Task publishArtifactsTask = getPublishArtifactsTask("$rootProject.ext.version", project)

This file was deleted.

This file was deleted.

Loading