Remove intermediate docker images with DOCKER_BUILDKIT=1 #9408

grubberr · 2022-01-11T10:28:14Z

Signed-off-by: Sergey Chvalyuk grubberr@gmail.com

What

Docker multi-stage build produce intermediate image which consume disk space.

./gradlew :airbyte-integrations:connectors:destination-redshift:airbyteDocker

docker images
airbyte/destination-redshift             dev                 796a4716004e   5 seconds ago   700MB
<none>                                   <none>              76409be86212   10 seconds ago   993MB

How

We build with DOCKER_BUILDKIT=1 to avoid intermediate images:

./gradlew :airbyte-integrations:connectors:destination-redshift:airbyteDocker

docker images
airbyte/destination-redshift             dev                 796a4716004e   5 seconds ago   700MB

🚨 User Impact 🚨

Are there any breaking changes? What is the end result perceived by the user? If yes, please merge this PR with the 🚨🚨 emoji so changelog authors can further highlight this if needed.

Pre-merge Checklist

Expand the relevant checklist and delete the others.

Updating a connector

Community member or Airbyter

Grant edit access to maintainers (instructions)
Secrets in the connector's spec are annotated with airbyte_secret
Unit & integration tests added and passing. Community members, please provide proof of success locally e.g: screenshot or copy-paste unit, integration, and acceptance test output. To run acceptance tests for a Python connector, follow instructions in the README. For java connectors run ./gradlew :airbyte-integrations:connectors:<name>:integrationTest.
Code reviews completed
Documentation updated
- Connector's README.md
- Connector's bootstrap.md. See description and examples
- Changelog updated in docs/integrations/<source or destination>/<name>.md including changelog. See changelog example
PR name follows PR naming conventions

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

Create a non-forked branch based on this PR and test the below items on it
Build is successful
Credentials added to Github CI. Instructions.
/test connector=connectors/<name> command is passing.
New Connector version released on Dockerhub by running the /publish command described here
After the new connector version is published, connector version bumped in the seed directory as described here
Seed specs have been re-generated by building the platform and committing the changes to the seed spec files, as described here

Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>

grubberr · 2022-01-11T10:50:17Z

/test connector=connectors/destination-redshift

🕑 connectors/destination-redshift https://github.com/airbytehq/airbyte/actions/runs/1682069326
✅ connectors/destination-redshift https://github.com/airbytehq/airbyte/actions/runs/1682069326
Python tests coverage:

	 ---------- coverage: platform linux, python 3.8.10-final-0 -----------
	 Name                                                              Stmts   Miss  Cover
	 -------------------------------------------------------------------------------------
	 main_dev_transform_catalog.py                                         3      3     0%
	 main_dev_transform_config.py                                          3      3     0%
	 normalization/__init__.py                                             4      0   100%
	 normalization/destination_type.py                                    13      0   100%
	 normalization/transform_catalog/__init__.py                           2      0   100%
	 normalization/transform_catalog/catalog_processor.py                143     77    46%
	 normalization/transform_catalog/destination_name_transformer.py     155      8    95%
	 normalization/transform_catalog/reserved_keywords.py                 13      0   100%
	 normalization/transform_catalog/stream_processor.py                 520    333    36%
	 normalization/transform_catalog/table_name_registry.py              174     34    80%
	 normalization/transform_catalog/transform.py                         45     26    42%
	 normalization/transform_catalog/utils.py                             33      7    79%
	 normalization/transform_config/__init__.py                            2      0   100%
	 normalization/transform_config/transform.py                         148     34    77%
	 -------------------------------------------------------------------------------------
	 TOTAL                                                              1258    525    58%

grubberr · 2022-01-11T10:51:50Z

/test connector=connectors/source-postgres

🕑 connectors/source-postgres https://github.com/airbytehq/airbyte/actions/runs/1682075792
❌ connectors/source-postgres https://github.com/airbytehq/airbyte/actions/runs/1682075792
🐛 https://gradle.com/s/susffudk7dmd6

grubberr · 2022-01-11T10:52:18Z

/test connector=connectors/destination-clickhouse

🕑 connectors/destination-clickhouse https://github.com/airbytehq/airbyte/actions/runs/1682077748
✅ connectors/destination-clickhouse https://github.com/airbytehq/airbyte/actions/runs/1682077748
No Python unittests run

grubberr · 2022-01-11T11:14:57Z

/test connector=connectors/destination-bigquery

🕑 connectors/destination-bigquery https://github.com/airbytehq/airbyte/actions/runs/1682170675
✅ connectors/destination-bigquery https://github.com/airbytehq/airbyte/actions/runs/1682170675
Python tests coverage:

	 ---------- coverage: platform linux, python 3.8.10-final-0 -----------
	 Name                                                              Stmts   Miss  Cover
	 -------------------------------------------------------------------------------------
	 main_dev_transform_catalog.py                                         3      3     0%
	 main_dev_transform_config.py                                          3      3     0%
	 normalization/__init__.py                                             4      0   100%
	 normalization/destination_type.py                                    13      0   100%
	 normalization/transform_catalog/__init__.py                           2      0   100%
	 normalization/transform_catalog/catalog_processor.py                143     77    46%
	 normalization/transform_catalog/destination_name_transformer.py     155      8    95%
	 normalization/transform_catalog/reserved_keywords.py                 13      0   100%
	 normalization/transform_catalog/stream_processor.py                 520    333    36%
	 normalization/transform_catalog/table_name_registry.py              174     34    80%
	 normalization/transform_catalog/transform.py                         45     26    42%
	 normalization/transform_catalog/utils.py                             33      7    79%
	 normalization/transform_config/__init__.py                            2      0   100%
	 normalization/transform_config/transform.py                         148     34    77%
	 -------------------------------------------------------------------------------------
	 TOTAL                                                              1258    525    58%

grubberr · 2022-01-11T11:16:26Z

/test connector=connectors/destination-cassandra

🕑 connectors/destination-cassandra https://github.com/airbytehq/airbyte/actions/runs/1682176434
✅ connectors/destination-cassandra https://github.com/airbytehq/airbyte/actions/runs/1682176434
No Python unittests run

grubberr · 2022-01-11T11:16:55Z

/test connector=connectors/destination-elasticsearch

🕑 connectors/destination-elasticsearch https://github.com/airbytehq/airbyte/actions/runs/1682178628
✅ connectors/destination-elasticsearch https://github.com/airbytehq/airbyte/actions/runs/1682178628
No Python unittests run

grubberr · 2022-01-11T11:17:12Z

/test connector=connectors/destination-gcs

🕑 connectors/destination-gcs https://github.com/airbytehq/airbyte/actions/runs/1682179593
✅ connectors/destination-gcs https://github.com/airbytehq/airbyte/actions/runs/1682179593
No Python unittests run

grubberr · 2022-01-11T11:17:26Z

/test connector=connectors/destination-kafka

🕑 connectors/destination-kafka https://github.com/airbytehq/airbyte/actions/runs/1682180504
✅ connectors/destination-kafka https://github.com/airbytehq/airbyte/actions/runs/1682180504
No Python unittests run

Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>

antixar

LGTM!
Please merge and bump version of SAT

Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>

grubberr · 2022-01-12T14:08:55Z

/test connector=source-acceptance-test

🕑 source-acceptance-test https://github.com/airbytehq/airbyte/actions/runs/1687977091
❌ source-acceptance-test https://github.com/airbytehq/airbyte/actions/runs/1687977091
🐛 https://gradle.com/s/5anwtml4zwnmm

grubberr · 2022-01-12T14:11:52Z

/test connector=bases/source-acceptance-test

🕑 bases/source-acceptance-test https://github.com/airbytehq/airbyte/actions/runs/1687990026
✅ bases/source-acceptance-test https://github.com/airbytehq/airbyte/actions/runs/1687990026
No Python unittests run

grubberr · 2022-01-12T14:18:41Z

/publish connector=bases/source-acceptance-test

🕑 bases/source-acceptance-test https://github.com/airbytehq/airbyte/actions/runs/1688020926
✅ bases/source-acceptance-test https://github.com/airbytehq/airbyte/actions/runs/1688020926

DOCKER_BUILDKIT=1 added

695e324

Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>

grubberr temporarily deployed to more-secrets January 11, 2022 10:29 Inactive

octavia-squidington-iii temporarily deployed to more-secrets January 11, 2022 10:52 Inactive

octavia-squidington-iii temporarily deployed to more-secrets January 11, 2022 10:53 Inactive

octavia-squidington-iii temporarily deployed to more-secrets January 11, 2022 10:54 Inactive

octavia-squidington-iii temporarily deployed to more-secrets January 11, 2022 11:16 Inactive

octavia-squidington-iii temporarily deployed to more-secrets January 11, 2022 11:18 Inactive

octavia-squidington-iii temporarily deployed to more-secrets January 11, 2022 11:19 Inactive

grubberr requested a review from antixar January 11, 2022 11:38

grubberr linked an issue Jan 11, 2022 that may be closed by this pull request

"Airbyte CI" docker multi-stage build consume too much disk space #9308

Closed

grubberr self-assigned this Jan 11, 2022

grubberr added 2 commits January 11, 2022 20:33

Merge branch 'master' into grubberr/9308-docker-buildkit

64a87e1

move DOCKER_BUILDKIT to top script, add docs

615efa0

Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>

grubberr temporarily deployed to more-secrets January 11, 2022 19:51 Inactive

antixar approved these changes Jan 12, 2022

View reviewed changes

grubberr added 2 commits January 12, 2022 16:05

Merge branch 'master' into grubberr/9308-docker-buildkit

a4466a0

source_acceptance_test 0.1.41 -> 0.1.42

1f933f5

Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>

grubberr temporarily deployed to more-secrets January 12, 2022 14:09 Inactive

octavia-squidington-iii temporarily deployed to more-secrets January 12, 2022 14:11 Inactive

octavia-squidington-iii temporarily deployed to more-secrets January 12, 2022 14:14 Inactive

octavia-squidington-iii temporarily deployed to more-secrets January 12, 2022 14:21 Inactive

grubberr merged commit 1581512 into master Jan 12, 2022

grubberr deleted the grubberr/9308-docker-buildkit branch January 12, 2022 14:28

grubberr mentioned this pull request Jan 12, 2022

Remove intermediate docker images by label 'io.airbyte.build=$BUILD_ID' #9309

Closed

40 tasks

karinakuz added connectors/destinations-api connectors/destination/elasticsearch labels Jan 13, 2022

octavia-squidington-iii mentioned this pull request Jan 13, 2022

Bump Airbyte version from 0.35.5-alpha to 0.35.6-alpha #9483

Closed

karinakuz added connectors/destinations-warehouse connectors/destination/bigquery connectors/destination/clickhouse connectors/destinations-streaming connectors/destination/kafka connectors/destination/cassandra connectors/destinations-database labels Jan 14, 2022

octavia-squidington-iii mentioned this pull request Jan 20, 2022

Bump Airbyte version from 0.35.5-alpha to 0.35.6-alpha #9630

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Remove intermediate docker images with DOCKER_BUILDKIT=1 #9408

Remove intermediate docker images with DOCKER_BUILDKIT=1 #9408

grubberr commented Jan 11, 2022 •

edited

Loading

grubberr commented Jan 11, 2022 •

edited by github-actions bot

Loading

grubberr commented Jan 11, 2022 •

edited by github-actions bot

Loading

grubberr commented Jan 11, 2022 •

edited by github-actions bot

Loading

grubberr commented Jan 11, 2022 •

edited by github-actions bot

Loading

grubberr commented Jan 11, 2022 •

edited by github-actions bot

Loading

grubberr commented Jan 11, 2022 •

edited by github-actions bot

Loading

grubberr commented Jan 11, 2022 •

edited by github-actions bot

Loading

grubberr commented Jan 11, 2022 •

edited by github-actions bot

Loading

antixar left a comment

grubberr commented Jan 12, 2022 •

edited by github-actions bot

Loading

grubberr commented Jan 12, 2022 •

edited by github-actions bot

Loading

grubberr commented Jan 12, 2022 •

edited by github-actions bot

Loading

Remove intermediate docker images with DOCKER_BUILDKIT=1 #9408

Remove intermediate docker images with DOCKER_BUILDKIT=1 #9408

Conversation

grubberr commented Jan 11, 2022 • edited Loading

What

How

Recommended reading order

🚨 User Impact 🚨

Pre-merge Checklist

Community member or Airbyter

Airbyter

grubberr commented Jan 11, 2022 • edited by github-actions bot Loading

grubberr commented Jan 11, 2022 • edited by github-actions bot Loading

grubberr commented Jan 11, 2022 • edited by github-actions bot Loading

grubberr commented Jan 11, 2022 • edited by github-actions bot Loading

grubberr commented Jan 11, 2022 • edited by github-actions bot Loading

grubberr commented Jan 11, 2022 • edited by github-actions bot Loading

grubberr commented Jan 11, 2022 • edited by github-actions bot Loading

grubberr commented Jan 11, 2022 • edited by github-actions bot Loading

antixar left a comment

Choose a reason for hiding this comment

grubberr commented Jan 12, 2022 • edited by github-actions bot Loading

grubberr commented Jan 12, 2022 • edited by github-actions bot Loading

grubberr commented Jan 12, 2022 • edited by github-actions bot Loading

grubberr commented Jan 11, 2022 •

edited

Loading

grubberr commented Jan 11, 2022 •

edited by github-actions bot

Loading

grubberr commented Jan 11, 2022 •

edited by github-actions bot

Loading

grubberr commented Jan 11, 2022 •

edited by github-actions bot

Loading

grubberr commented Jan 11, 2022 •

edited by github-actions bot

Loading

grubberr commented Jan 11, 2022 •

edited by github-actions bot

Loading

grubberr commented Jan 11, 2022 •

edited by github-actions bot

Loading

grubberr commented Jan 11, 2022 •

edited by github-actions bot

Loading

grubberr commented Jan 11, 2022 •

edited by github-actions bot

Loading

grubberr commented Jan 12, 2022 •

edited by github-actions bot

Loading

grubberr commented Jan 12, 2022 •

edited by github-actions bot

Loading

grubberr commented Jan 12, 2022 •

edited by github-actions bot

Loading