Disable cache if retried 3 times

[ryoqun]

Problem

There's no way for team eng to get out of being stuck after ci cache issue. Also, there's no way to disable it for debugging as well.

Summary of Changes

implement ala circleci's build-without-cache feature under very limited ui functionality we have at buildkite => hit retry 3 times for third time luck boost. lol

seems to work:

[yihau]

afaik we have two kinds of cache. 1 is the local one which stored in the ~/.cargo and 2 is sccache.
iirc disable sccache should be enough to solve the issue. maybe we can do something like

if [[ -n $CI ]]; then
   # Share the real ~/.cargo between docker containers in CI for speed
   ARGS+=(--volume "$HOME:/home")

   if [[ -n $BUILDKITE ]]; then
-    # sccache
-    ARGS+=(
-      --env "RUSTC_WRAPPER=/home/.cargo/bin/sccache"
-      --env AWS_ACCESS_KEY_ID
-      --env AWS_SECRET_ACCESS_KEY
-      --env SCCACHE_BUCKET
-      --env SCCACHE_REGION
-    )
+    # Fully disable caching to create a clean-room environment to preclude any
+    # cache-related bugs for our sanity
+    if [[ $BUILDKITE_RETRY_COUNT -ge 3 ]]; then
+      # I hate buildkite-esque echo is leaking into this generic shell wrapper.
+      # but it's easiest, in edge-case, and properly guarded under $BUILDKITE_ env...
+      echo "--- DISABLING SCCACHE DUE TO MANY (${BUILDKITE_RETRY_COUNT}) RETRIES"
+    else
+      # sccache
+      ARGS+=(
+        --env "RUSTC_WRAPPER=/home/.cargo/bin/sccache"
+        --env AWS_ACCESS_KEY_ID
+        --env AWS_SECRET_ACCESS_KEY
+        --env SCCACHE_BUCKET
+        --env SCCACHE_REGION
+      )
+    fi
   fi
 else
   # Avoid sharing ~/.cargo when building locally to avoid a mixed macOS/Linux
fi
ARGS+=(--env "HOME=/home" --env "CARGO_HOME=/home/.cargo")

[ryoqun]

afaik we have two kinds of cache. 1 is the local one which stored in the ~/.cargo and 2 is sccache. iirc disable sccache should be enough to solve the issue. maybe we can do something like

if [[ -n $CI ]]; then
   # Share the real ~/.cargo between docker containers in CI for speed
   ARGS+=(--volume "$HOME:/home")

   if [[ -n $BUILDKITE ]]; then
-    # sccache
-    ARGS+=(
-      --env "RUSTC_WRAPPER=/home/.cargo/bin/sccache"
-      --env AWS_ACCESS_KEY_ID
-      --env AWS_SECRET_ACCESS_KEY
-      --env SCCACHE_BUCKET
-      --env SCCACHE_REGION
-    )
+    # Fully disable caching to create a clean-room environment to preclude any
+    # cache-related bugs for our sanity
+    if [[ $BUILDKITE_RETRY_COUNT -ge 3 ]]; then
+      # I hate buildkite-esque echo is leaking into this generic shell wrapper.
+      # but it's easiest, in edge-case, and properly guarded under $BUILDKITE_ env...
+      echo "--- DISABLING SCCACHE DUE TO MANY (${BUILDKITE_RETRY_COUNT}) RETRIES"
+    else
+      # sccache
+      ARGS+=(
+        --env "RUSTC_WRAPPER=/home/.cargo/bin/sccache"
+        --env AWS_ACCESS_KEY_ID
+        --env AWS_SECRET_ACCESS_KEY
+        --env SCCACHE_BUCKET
+        --env SCCACHE_REGION
+      )
+    fi
   fi
 else
   # Avoid sharing ~/.cargo when building locally to avoid a mixed macOS/Linux
fi
ARGS+=(--env "HOME=/home" --env "CARGO_HOME=/home/.cargo")

@yihau thanks for the suggestion. I've took your advance with some tweak. could you take a look at it?

[yihau]

❤️

[ryoqun]

this message will be shown always for easier user understanding from consistent behavior: