Skip to content

Commit

Permalink
pathogen-repo-build: Move build.log to ignored directory
Browse files Browse the repository at this point in the history
The entire working directory gets zipped and uploaded to AWS Batch when
using the `aws-batch` runtime, which includes an early `build.log` file.
When the build completes and downloads the build outputs, the early
`build.log` overwrites the local `build.log` of the GH Action.
This results in a truncated file that does not include the
AWS_BATCH_JOB_ID and prevents the generation of the AWS Batch summary.

This commit moves the hard-coded `build.log` file to an ignored directory
(the same directory that we are using for the nextstrain/.github repo¹).
This prevents the `build.log` from being uploaded with the build.

Previously tried other paths, but they all caused errors.
- "../build.log" - ".." pattern is not allowed for the `upload-artifact` action²
- "~/build.log" - `hashfiles` requires files to be in the `GITHUB_WORKSPACE`³

¹ #44 (comment)
² actions/upload-artifact#176
³ https://docs.github.com/en/actions/learn-github-actions/expressions#hashfiles
  • Loading branch information
joverlee521 committed Mar 15, 2024
1 parent 5cd0ba1 commit 5c5a528
Show file tree
Hide file tree
Showing 2 changed files with 20 additions and 17 deletions.
19 changes: 10 additions & 9 deletions .github/workflows/pathogen-repo-build.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -98,14 +98,14 @@ on:
The default paths included in the artifact are:
build.log
.git/build.log
auspice/
results/
benchmarks/
logs/
.snakemake/log/
The "build.log" contains log messages from the `nextstrain build` command. The other paths are common output paths for Nextstrain builds. If a path does not exist in your build, then the action will still succeed and will print out a warning for the non-existent file(s). Use an exclude pattern for any of the default paths that you would like to exclude from the artifact (e.g. !build.log).
The ".git/build.log" contains log messages from the `nextstrain build` command. The other paths are common output paths for Nextstrain builds. If a path does not exist in your build, then the action will still succeed and will print out a warning for the non-existent file(s). Use an exclude pattern for any of the default paths that you would like to exclude from the artifact (e.g. !.git/build.log).
This is not supported for builds on AWS Batch because the workflow detaches from the build. Please use the `nextstrain build` command locally to reattach to AWS Batch builds to download outputs.
type: string
Expand Down Expand Up @@ -189,14 +189,14 @@ on:
The default paths included in the artifact are:
build.log
.git/build.log
auspice/
results/
benchmarks/
logs/
.snakemake/log/
The "build.log" contains log messages from the `nextstrain build` command. The other paths are common output paths for Nextstrain builds. If a path does not exist in your build, then the action will still succeed and will print out a warning for the non-existent file(s). Use an exclude pattern for any of the default paths that you would like to exclude from the artifact (e.g. !build.log).
The ".git/build.log" contains log messages from the `nextstrain build` command. The other paths are common output paths for Nextstrain builds. If a path does not exist in your build, then the action will still succeed and will print out a warning for the non-existent file(s). Use an exclude pattern for any of the default paths that you would like to exclude from the artifact (e.g. !.git/build.log).
This is not supported for builds on AWS Batch because the workflow detaches from the build. Please use the `nextstrain build` command locally to reattach to AWS Batch builds to download outputs.
type: string
Expand All @@ -209,6 +209,7 @@ on:
required: true
env:
NEXTSTRAIN_GITHUB_DIR: .git/nextstrain/.github
NEXTSTRAIN_BUILD_LOG: .git/build.log
permissions:
id-token: write
jobs:
Expand Down Expand Up @@ -274,14 +275,14 @@ jobs:
# shellcheck disable=SC2154
set -x
eval "$NEXTSTRAIN_BUILD_COMMAND" |& tee build.log
eval "$NEXTSTRAIN_BUILD_COMMAND" |& tee "$NEXTSTRAIN_BUILD_LOG"
# Attempt to get the AWS Batch ID even if the run build command failed
# as long as the runtime is `aws-batch` and the `build.log` file exists
- if: ${{ always() && inputs.runtime == 'aws-batch' && hashFiles('build.log') != '' }}
# as long as the runtime is `aws-batch` and the `NEXTSTRAIN_BUILD_LOG` file exists
- if: ${{ always() && inputs.runtime == 'aws-batch' && hashFiles(env.NEXTSTRAIN_BUILD_LOG) != '' }}
name: Get AWS Batch job id
id: aws-batch
run: |
echo "AWS_BATCH_JOB_ID=$(sed -nE 's/.+AWS Batch Job ID\:.+ ([-a-f0-9]+)$/\1/p' < build.log)" | tee -a "$GITHUB_ENV"
echo "AWS_BATCH_JOB_ID=$(sed -nE 's/.+AWS Batch Job ID\:.+ ([-a-f0-9]+)$/\1/p' < $NEXTSTRAIN_BUILD_LOG)" | tee -a "$GITHUB_ENV"
- if: ${{ always() && env.AWS_BATCH_JOB_ID }}
name: Generate AWS Batch summary
run: |
Expand All @@ -293,7 +294,7 @@ jobs:
if-no-files-found: warn
name: ${{ inputs.artifact-name }}
path: |
build.log
${{ env.NEXTSTRAIN_BUILD_LOG }}
auspice/
results/
benchmarks/
Expand Down
18 changes: 10 additions & 8 deletions .github/workflows/pathogen-repo-build.yaml.in
Original file line number Diff line number Diff line change
Expand Up @@ -128,19 +128,20 @@ on:

The default paths included in the artifact are:

build.log
.git/build.log
auspice/
results/
benchmarks/
logs/
.snakemake/log/

The "build.log" contains log messages from the `nextstrain build` command.
The ".git/build.log" contains log messages from the
`nextstrain build` command.
The other paths are common output paths for Nextstrain builds.
If a path does not exist in your build, then the action will still
succeed and will print out a warning for the non-existent file(s).
Use an exclude pattern for any of the default paths that you would like to
exclude from the artifact (e.g. !build.log).
exclude from the artifact (e.g. !.git/build.log).

This is not supported for builds on AWS Batch because the workflow
detaches from the build. Please use the `nextstrain build` command
Expand All @@ -160,6 +161,7 @@ on:

env:
NEXTSTRAIN_GITHUB_DIR: .git/nextstrain/.github
NEXTSTRAIN_BUILD_LOG: .git/build.log

permissions:
id-token: write
Expand Down Expand Up @@ -240,15 +242,15 @@ jobs:
# shellcheck disable=SC2154
set -x

eval "$NEXTSTRAIN_BUILD_COMMAND" |& tee build.log
eval "$NEXTSTRAIN_BUILD_COMMAND" |& tee "$NEXTSTRAIN_BUILD_LOG"

# Attempt to get the AWS Batch ID even if the run build command failed
# as long as the runtime is `aws-batch` and the `build.log` file exists
- if: ${{ always() && inputs.runtime == 'aws-batch' && hashFiles('build.log') != '' }}
# as long as the runtime is `aws-batch` and the `NEXTSTRAIN_BUILD_LOG` file exists
- if: ${{ always() && inputs.runtime == 'aws-batch' && hashFiles(env.NEXTSTRAIN_BUILD_LOG) != '' }}
name: Get AWS Batch job id
id: aws-batch
run: |
echo "AWS_BATCH_JOB_ID=$(sed -nE 's/.+AWS Batch Job ID\:.+ ([-a-f0-9]+)$/\1/p' < build.log)" | tee -a "$GITHUB_ENV"
echo "AWS_BATCH_JOB_ID=$(sed -nE 's/.+AWS Batch Job ID\:.+ ([-a-f0-9]+)$/\1/p' < $NEXTSTRAIN_BUILD_LOG)" | tee -a "$GITHUB_ENV"

- if: ${{ always() && env.AWS_BATCH_JOB_ID }}
name: Generate AWS Batch summary
Expand All @@ -262,7 +264,7 @@ jobs:
if-no-files-found: warn
name: ${{ inputs.artifact-name }}
path: |
build.log
${{ env.NEXTSTRAIN_BUILD_LOG }}
auspice/
results/
benchmarks/
Expand Down

0 comments on commit 5c5a528

Please sign in to comment.