[discussion] [docker] Multiple uploaded artifacts & matrix strategy for docker-based

[asraa]

This issue is to discuss the output artifacts for the docker-based workflow.

The docker-based workflow allows wildcard artifact paths, say dist/** which may match a number of artifacts and subfolders, for e.g. dist/subfolder/b.txt and dist/a.txt.

When the workflow is complete, they are currently stored in GH artifacts in their subfolders, using https://github.com/actions/upload-artifact#upload-using-multiple-paths-and-exclusions

This preserves the path heirarchy after the first wildcard pattern.

On a GH tag or release event, https://github.com/softprops/action-gh-release supports glob patterns, so we can upload all the artifacts. They will be uploaded as individual files.

Question 1 There is a bit of a mismatch between uploading the path-preserved files in the GH artifacts, and uploading the individual files. Should we remove the path heirarchy preservation and assume that generated artifacts do not need that structure?

Question 2 When using matrix strategy or multiple runs, I believe the build output and provenance folders should be separated in the GH artifact uploads in the workflow. They will be things like build-outputs-RNG_A and build-outputs-RNG_B for each of the matrix strategies. This way you can distinguish between each job in the matrices outputs. Again, for a release, they will all be uploaded as a flat structure to the release. Does that make sense?

cc @rbehjati

[rbehjati]

Sorry for the late reply.

Question 1 There is a bit of a mismatch between uploading the path-preserved files in the GH artifacts, and uploading the individual files. Should we remove the path heirarchy preservation and assume that generated artifacts do not need that structure?

I think we cannot generally assume this. Removing the structure will work with the current setup of Oak, but I think that can in general be an annoying restriction for the user.

Question 2 When using matrix strategy or multiple runs, I believe the build output and provenance folders should be separated in the GH artifact uploads in the workflow. They will be things like build-outputs-RNG_A and build-outputs-RNG_B for each of the matrix strategies. This way you can distinguish between each job in the matrices outputs.

This part makes sense to me, and seems to work fine.

Again, for a release, they will all be uploaded as a flat structure to the release. Does that make sense?

Does a flat structure mean that files with the same base name will be overridden?
If that is the case, I think the best thing we can do it to be clear about this behaviour, so that the users configure their artifact_paths accordingly. I think the artifact_path field in the config file gives enough flexibility.