[Bug]: Images built not saved in the local image store

[celeste-zeng]

What happened?

Hi,

I tried to build a Beam Go SDK container by running ./gradlew :sdks:go:container:docker. The build succeeded but when I do docker image ls or docker images, I couldn't find the image built and saw WARNING: No output specified with docker-container driver. Build result will only remain in the build cache. To push result image into registry use --push or to load image into docker use --load. So though the build is finished, the image built is not saved and disappeared.

Based on what I found here: https://docs.docker.com/build/drivers/docker-container/, this is because it is using the docker-container driver and "unlike when using the default docker driver, images built with the docker-container driver must be explicitly loaded into the local image store. Use the --load flag."

Therefore, I tried to add load true as the last line in the docker task in sdks/go/container/build.gradle and ran the same build command again. Then I was able to find the apache/beam_go_sdk container locally. The same issue exists for all Beam Python and Java SDK containers. And adding load true as the last line in the docker task in sdks/$LANGUAGE/container/common.gradle will resolve it.

However, one other thing worth noting is that if we want to build multi-arch images by adding, for example,-Pcontainer-architecture-list=arm64,amd64 to the build command, load true will make the build fail because ERROR: docker exporter does not currently support exporting manifest lists to local image store.

An alternative solution will be adding push true (which is mutually exclusive with load true) and it will automatically push the image built to the specified repository. A multi-arch image can be built and pushed this way. If no repository is specified, the build will fail because it will by default try to push the image to the apache repository, to which we don't have permission.

Conclusively, we can make the below changes in the corresponding common.gradle and build.gradle files:

1. if multiple architectures are passed as argument, set push true, build and push the image to the specified repository and if no repository specified, throw an informational error such as ERROR: docker exporter does not currently support exporting manifest lists, to save multi-arch images, please specify a repository to push;
2. otherwise set load true and always load the image built to local image store by default.

Any other suggestions are welcomed, thanks!

Issue Priority

Priority: 1 (data loss / total loss of function)

Issue Components

• Component: Python SDK
• Component: Java SDK
• Component: Go SDK
• Component: Typescript SDK
• Component: IO connector
• Component: Beam examples
• Component: Beam playground
• Component: Beam katas
• Component: Website
• Component: Spark Runner
• Component: Flink Runner
• Component: Samza Runner
• Component: Twister2 Runner
• Component: Hazelcast Jet Runner
• Component: Google Cloud Dataflow Runner

[celeste-zeng]

.take-issue

[lostluck]

cc: @robertwb due to multi-arch docker images

[robertwb]

Given the current limitations, I'm in favor of pushing if multi-arch builds are requested. (I would not want to push by default, relying on the fact that most (but not all) people don't have push permissions to the beam repos, but if we require an explict repository when multi-arch is requested that seems reasonable to me.)

[kennknowles]

Do I understand correctly that the build is literally useless, because the artifact is immediately discarded?

Or is there some way that the artifact turns out to be used in downstream gradle builds?

[celeste-zeng]

Sorry for the late reply. I think the artifact is discarded and can't be found.