Skip to content

Commit cfcc8e3

Browse files
committed
entrypoint-aws-batch: Remove existing /nextstrain/{augur,auspice,fauna}/ for overlays
Makes these overlays work the same on AWS Batch as they do in local containers (Docker or Singularity): the overlays fully replace the image's copy. This addresses a caveat noted the Nextstrain CLI commit adding AWS Batch overlay support¹ that was further discussed in review.² ZIP archive directory members always end in "/", so we test only for the presence of ../{augur,auspice,fauna}/ in the archive. This works consistently for workdir archives with overlays produced by Nextstrain CLI, but it isn't foolproof for archives produced other ways. For example, it is possible for a ../augur/x/y/z path to be present in the archive without ../augur/ also being present. In that case, nothing will be deleted. This could be seen as a bug or a feature. ¹ <nextstrain/cli@99168ac> ² <nextstrain/cli#419 (comment)>
1 parent a947a79 commit cfcc8e3

4 files changed

+87
-0
lines changed

entrypoint-aws-batch

+4
Original file line numberDiff line numberDiff line change
@@ -10,6 +10,10 @@ fi
1010
case "$NEXTSTRAIN_AWS_BATCH_WORKDIR_URL" in
1111
s3://*.zip)
1212
aws s3 cp --no-progress "$NEXTSTRAIN_AWS_BATCH_WORKDIR_URL" "$PWD.zip"
13+
(zipinfo -1 "$PWD.zip" ../{augur,auspice,fauna}/ 2>/dev/null || true) | while read -r dir; do
14+
echo "removing $dir for overlay"
15+
rm -rf "$dir"
16+
done
1317
unzip -: -o "$PWD.zip"
1418
;;
1519
s3://*)

tests/aws-batch-workdir-url.t

+83
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,83 @@
1+
#!/usr/bin/env cram
2+
3+
Setup.
4+
5+
$ [[ -n "$AWS_ACCESS_KEY_ID" && -n "$AWS_SECRET_ACCESS_KEY" ]] || exit 80
6+
7+
$ : "${IMAGE:=localhost:5000/nextstrain/base:latest}"
8+
$ (docker image inspect "$IMAGE" || docker image pull "$IMAGE") &>/dev/null
9+
10+
$ export NEXTSTRAIN_AWS_BATCH_VERBOSE=0
11+
12+
Workdir ZIP archive is downloaded and extracted.
13+
14+
$ export NEXTSTRAIN_AWS_BATCH_WORKDIR_URL="s3://nextstrain-tmp/$(python3 -c 'import uuid; print(uuid.uuid4())').zip"
15+
16+
$ aws s3 cp --quiet "$TESTDIR/data/workdir-without-overlays.zip" "$NEXTSTRAIN_AWS_BATCH_WORKDIR_URL"
17+
18+
$ docker run --rm --env=NEXTSTRAIN_AWS_BATCH_{WORKDIR_URL,VERBOSE} --env=AWS_{ACCESS_KEY_ID,SECRET_ACCESS_KEY,SESSION_TOKEN} "$IMAGE" \
19+
> /sbin/entrypoint-aws-batch bash -euo pipefail -c 'ls -l'
20+
download: s3://nextstrain-tmp/*.zip to ../build.zip (glob)
21+
Archive: /nextstrain/build.zip
22+
extracting: reticulating
23+
extracting: splines
24+
total 0
25+
-rw-rw-r-- 1 nextstrain nextstrain 0 Mar 10 21:46 reticulating
26+
-rw-rw-r-- 1 nextstrain nextstrain 0 Mar 10 21:46 splines
27+
upload: ../build.zip to s3://nextstrain-tmp/*.zip (glob)
28+
29+
Overlays are handled.
30+
31+
$ export NEXTSTRAIN_AWS_BATCH_WORKDIR_URL="s3://nextstrain-tmp/$(python3 -c 'import uuid; print(uuid.uuid4())').zip"
32+
33+
$ aws s3 cp --quiet "$TESTDIR/data/workdir-with-augur-auspice-overlays.zip" "$NEXTSTRAIN_AWS_BATCH_WORKDIR_URL"
34+
35+
$ docker run --rm --env=NEXTSTRAIN_AWS_BATCH_{WORKDIR_URL,VERBOSE} --env=AWS_{ACCESS_KEY_ID,SECRET_ACCESS_KEY,SESSION_TOKEN} "$IMAGE" \
36+
> /sbin/entrypoint-aws-batch bash -euo pipefail -c 'ls -lR . ../augur ../auspice'
37+
download: s3://nextstrain-tmp/*.zip to ../build.zip (glob)
38+
removing ../augur/ for overlay
39+
removing ../auspice/ for overlay
40+
Archive: /nextstrain/build.zip
41+
extracting: reticulating
42+
extracting: splines
43+
creating: ../augur/
44+
creating: ../augur/a/
45+
creating: ../augur/a/b/
46+
creating: ../augur/a/b/c/
47+
extracting: ../augur/a/b/c/world.txt
48+
extracting: ../augur/a/b/c/hello.txt
49+
creating: ../augur/augur/
50+
extracting: ../augur/augur/__init__.py
51+
extracting: ../augur/README.md
52+
creating: ../auspice/
53+
.:
54+
total 0
55+
-rw-rw-r-- 1 nextstrain nextstrain 0 Mar 10 21:46 reticulating
56+
-rw-rw-r-- 1 nextstrain nextstrain 0 Mar 10 21:46 splines
57+
58+
../augur:
59+
total 12
60+
-rw-rw-r-- 1 nextstrain nextstrain 22 Mar 10 21:45 README.md
61+
drwxrwxr-x 3 nextstrain nextstrain 4096 Mar 10 21:32 a
62+
drwxrwxr-x 2 nextstrain nextstrain 4096 Mar 10 21:46 augur
63+
64+
../augur/a:
65+
total 4
66+
drwxrwxr-x 3 nextstrain nextstrain 4096 Mar 10 21:32 b
67+
68+
../augur/a/b:
69+
total 4
70+
drwxrwxr-x 2 nextstrain nextstrain 4096 Mar 10 21:33 c
71+
72+
../augur/a/b/c:
73+
total 8
74+
-rw-rw-r-- 1 nextstrain nextstrain 6 Mar 10 21:32 hello.txt
75+
-rw-rw-r-- 1 nextstrain nextstrain 6 Mar 10 21:32 world.txt
76+
77+
../augur/augur:
78+
total 4
79+
-rw-rw-r-- 1 nextstrain nextstrain 34 Mar 10 21:46 __init__.py
80+
81+
../auspice:
82+
total 0
83+
upload: ../build.zip to s3://nextstrain-tmp/*.zip (glob)
Binary file not shown.
292 Bytes
Binary file not shown.

0 commit comments

Comments
 (0)