Upload artifacts and logs to S3 bucket #114
Conversation
marbre
force-pushed
the
upload-artifacts
branch
6 times, most recently
from
694b322 to
87e3a92
Compare
* Uses the `configure-aws-credentials` action to retrieve short-lived credentials using GitHub's OIDC provider * Creates index of files using the MIT-licensed indexer.py script * Uploads artifacts and logs to the S3 bucket * Adds links pointing to the bucket to the job summary Creating index files and uploading files is to be moved to scripts in a follow up.
marbre
force-pushed
the
upload-artifacts
branch
from
51616e9 to
e17fdc5
Compare
marbre
changed the title
|
See for example https://github.com/nod-ai/TheRock/actions/runs/13532379300 for a run limited to build |
|
Artifacts were successfully upload (https://therock-artifacts.s3.us-east-2.amazonaws.com/13532940414/index.html) but seems logs were not. Looking into it. |
marbre
force-pushed
the
upload-artifacts
branch
from
58878d3 to
8798221
Compare
marbre
force-pushed
the
upload-artifacts
branch
from
611360a to
3a229fe
Compare
marbre
force-pushed
the
upload-artifacts
branch
from
92b3022 to
3c8c12f
Compare
|
Fixed the issue. |
| uses: aws-actions/configure-aws-credentials@ececac1a45f3b08a01d2dd070d28d111c5fe6722 # v4.1.0 | ||
| with: | ||
| aws-region: us-east-2 | ||
| role-to-assume: arn:aws:iam::692859939525:role/therock-artifacts |
There was a problem hiding this comment.
For later: should we have this in cleartext like this? Not just security, but forks will get it. If a secret, the you could also make the step conditional on the secret existing.
There was a problem hiding this comment.
This is not a secret and just a role to assume. This role is used to fetch credentials and is configured in AWS IAM to be only accessible from the nod-ai/TheRock repository. Not even another repository within our org is able to assume it. Happy to provide further details.
There was a problem hiding this comment.
Yes, I was more suggesting that storing it in a secret may be a convenient mechanism to make this whole thing fork tolerant: have a key that you can condition any of the AWS steps on only if it is not empty.
The what happens in a fork thing is the main thing I was poking on. What happens if a run fails to get this credential? Whole run fails (vs no uploading happens)?
There was a problem hiding this comment.
Ah, got your point now. Happy to improve that.
| --exclude "*" \ | ||
| --include "*.log" --include "index.html" | ||
|
|
||
| - name: Add Links to Job Summary |
| role-to-assume: arn:aws:iam::692859939525:role/therock-artifacts | ||
|
|
||
| # TODO: Move to script | ||
| - name: Create Index Files |
There was a problem hiding this comment.
If we move this into the build, we're going to want another was to collect the index. Probably by having specially named marker files for each entry in a logs subdir (because each action that writes one will need to be able to add a file without stepping on anyone else).
Alternatively, we could have every one "stream_artifacts" dir, and if files go in there, they get uploaded. If the script you write was able to run in the background, you could just start it at the beginning of a run and have it loop/run to the end conditioned on something like:
while not done():
Where done returns true if there isn't a .done file. And then it would upload each file that shows up in that directory. At the end of a run, you would write the .done file, then wait, and run it one more time to get any last bits that weren't found in the background. You'd need to manage a couple more marker files to let it run multiple times without reuploading.
If you did that, would be better to advertise the location link at the beginning of the job and write an updated index page on each iteration.
Thanks for the pointers, I will gladly take them into account! |
| # TODO: Move to script | ||
| - name: Upload Artifacts | ||
| run: | | ||
| aws s3 cp build/artifacts/ s3://therock-artifacts/${{github.run_id}}/ \ |
There was a problem hiding this comment.
you'll probably want to add a "linux" prefix to this path once we start uploading windows artifacts.
There was a problem hiding this comment.
Good point! While the github.run_id is unique for each workflow run within a repository we right now have a single workflow defined in the ci.yml which then triggers for the different systems. Would you prefer a prefix over a suffix? Or maybe even a separate S3 bucket for Windows artifacts?
There was a problem hiding this comment.
not opinionated on prefix vs suffix. I suggest we only do different s3 buckets for different data retention policies (for example release builds vs ci builds etc). I expect we will be adding additional suffixes for variants and such over time.
|
Left a quick comment. Thanks for getting this in - makes sense! |
configure-aws-credentialsaction to retrieve short-livedcredentials using GitHub's OIDC provider
Creating index files and uploading files is to be moved to scripts in a
follow up.