Skip to content

Commit 7eaf158

Browse files
jleibsjleibs
authored
Introduce new reusable workflow jobs and cleanup manual trigger (#1954)
There are 8 reusable workflow "components" that we can use to build different scenarios: reusable_checks.yml - These are all the checks that run to ensure the code is formatted, reusable_bench.yml - This job runs the benchmarks to check for performance regressions. reusable_deploy_docs- This job deploys the python and rust documentation to https://ref.rerun.io reusable_build_and_test_wheels.yml - This job builds the wheels, runs the end-to-end test, and produces a sample RRD. The artifacts are accessible via GitHub artifacts, but not otherwise uploaded anywhere. reusable_upload_wheels.yml- This job uploads the wheels to google cloud reusable_build_web.yml - This job builds the wasm artifacts for the web. reusable_upload_web.yml - This job uploads the web assets to google cloud. By default this only uploads to: app.rerun.io/commit/<commit>/ reusable_pr_summary.yml - This job updates the PR summary with the results of the CI run. Example summary can be found at: https://storage.googleapis.com/rerun-builds/pull_request/1954/index.html This also introduces a manual_dispatch.yml helper as a convenience for testing these workflows and their different parameterizations.
1 parent 840a127 commit 7eaf158

18 files changed

+1734
-103
lines changed

‎.github/workflows/README.md

+66
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,66 @@
1+
# Overview
2+
3+
Our CI workflows make heavy usage of [Reusable Workflows](https://docs.github.com/en/actions/using-workflows/reusing-workflows). These reusable workflows can then be tested manually via the `manual_dispatch.yml` workflow.
4+
Or integrated into CI jobs such has `on_pull_request.yml` or `on_main.yml`.
5+
6+
By convention:
7+
- All reusable workflows start with the `reusable_` prefix.
8+
- All workflows that are triggered via `workflow_dispatch` start with the `manual_` prefix.
9+
- All workflows that are triggered via an event start with the `on_` prefix.
10+
- `on_pull_request` is triggered on pull requests.
11+
- `on_main` is triggered on pushes to the main branch.
12+
13+
If you are going to be doing any editing of workflows, the
14+
[VS Code extension](https://marketplace.visualstudio.com/items?itemName=cschleiden.vscode-github-actions)
15+
for GitHub Actions is highly recommended.
16+
17+
## Reusable Workflows
18+
- [reusable_checks.yml](reusable_checks.yml) - These are all the checks that run to ensure the code is formatted,
19+
linted, and tested. This job produces no artifacts other than a pass/fail criteria for the build.
20+
- `SAVE_CACHE` - If true, the rust cache will be saved. Generally we only do this for builds on `main`
21+
- [reusable_bench.yml](reusable_bench.yml) - This job runs the benchmarks to check for performance regressions.
22+
- `SAVE_BENCH` - If true, then the benchmark results are saved to update https://ref.rerun.io/dev/bench/
23+
- [reusable_deploy_docs](reusable_deploy_docs.yml) - This job deploys the python and rust documentation to https://ref.rerun.io
24+
- `PY_DOCS_VERSION_NAME` - The name to use for versioning the python docs. This should generally match the version in
25+
`Cargo.toml`.
26+
- `UPDATE_LATEST` - If true, then the docs will be deployed to `latest/` as well as the versioned directory.
27+
- [reusable_build_and_test_wheels.yml](reusable_build_and_test_wheels.yml) - This job builds the wheels, runs the
28+
end-to-end test, and produces a sample RRD. The artifacts are accessible via GitHub artifacts, but not otherwise
29+
uploaded anywhere.
30+
- `MATURIN_FEATURE_FLAGS` - The feature flags to pass to maturin.
31+
- `PLATFORM` - Which platform to build for: `linux`, `macos-arm`, `macos-intel`, or `windows`.
32+
- `RELEASE_VERSION` - If producing a release, the version number. This must match the version in `Cargo.toml`.
33+
- `RRD_ARTIFACT_NAME` - Intermediate name of the GitHub rrd artifact for passing to `reusable_upload_wheels.yml`
34+
- `SAVE_CACHE` - If true, the rust cache will be saved. Generally we only do this for builds on `main`
35+
- `WHEEL_ARTIFACT_NAME` - Intermediate name of the GitHub wheel artifact for passing to `reusable_upload_wheels.yml`
36+
- [reusable_upload_wheels.yml](reusable_upload_wheels.yml) - This job uploads the wheels to google cloud
37+
- `RRD_ARTIFACT_NAME` - Intermediate name of the GitHub rrd artifact. This should match the name passed to
38+
`reusable_build_and_test_wheels.yml`
39+
- `WHEEL_ARTIFACT_NAME` - Intermediate name of the GitHub wheel artifact. This should match the name passed to
40+
`reusable_build_and_test_wheels.yml`
41+
- [reusable_build_web.yml](reusable_build_web.yml) - This job builds the wasm artifacts for the web.
42+
- `RELEASE_VERSION` - If producing a release, the version number. This must match the version in `Cargo.toml`.
43+
- [reusable_upload_web.yml](reusable_upload_web.yml) - This job uploads the web assets to google cloud. By default this
44+
only uploads to: `app.rerun.io/commit/<commit>/`
45+
- `MARK_PRERELEASE_FOR_MAINLINE` - If true, then the web assets will go to `app.rerun.io/preleease/
46+
- `MARK_TAGGED_VERSION` - If true, then the web assets will go to `app.rerun.io/version/<RELEASE_VERSION>`
47+
- `RELEASE_VERSION` - If producing a release, the version number.
48+
- `RRD_ARTIFACT_NAME` - Intermediate name of the GitHub rrd artifact. This should match the name passed to
49+
`reusable_build_and_test_wheels.yml`
50+
- `UPLOAD_COMMIT_OVERRIDE` - If set, will replace the value of `<commit>`. This is necessary because we want pull
51+
request builds associated with their originating commit, even if the web-build happens on an ephemeral merge-commit.
52+
- [reusable_pr_summary.yml](reusable_pr_summary.yml) - This job updates the PR summary with the results of the CI run.
53+
- This summary can be found at:
54+
`https://storage.googleapis.com/rerun-builds/pull_request/<PR_NUMBER>/index.html`
55+
- `PR_NUMBER` - The PR number to update. This will generally be set by the `on_pull_request.yml` workflow using:
56+
`${{github.event.pull_request.number}}`
57+
58+
## Manual Workflows
59+
- [manual_dispatch](manual_dispatch.yml) - This workflow is used to manually trigger the assorted reusable workflows for
60+
testing.
61+
- See the workflow file for the list of parameters.
62+
- [manual_build_wheels_for_pr.yml](manual_build_wheels_for_pr.yml) - This workflow can be dispatched on a branch and
63+
will build all of the wheels for the associated pull-request. Uses:
64+
- [reusable_build_and_test_wheels.yml](reusable_build_and_test_wheels.yml)
65+
- [reusable_upload_wheels.yml](reusable_upload_wheels.yml)
66+
- [reusable_pr_summary.yml](reusable_pr_summary.yml)

‎.github/workflows/manual_build_wheels_for_pr.yml

+86-2
Original file line numberDiff line numberDiff line change
@@ -5,7 +5,7 @@ on:
55

66
jobs:
77

8-
check_for_pr:
8+
check-for-pr:
99
runs-on: ubuntu-latest
1010
outputs:
1111
PR_NUMBER: ${{ steps.get_pr.outputs.PR_NUMBER }}
@@ -24,5 +24,89 @@ jobs:
2424
exit 1
2525
else
2626
echo "Commit is associated with PR: $pr_number"
27-
echo "PR_NUMBER=$pr_number" >> "$GITHUB_OUTPUTS"
27+
echo "PR_NUMBER=$pr_number" >> "$GITHUB_OUTPUT"
2828
fi
29+
30+
build-linux:
31+
needs: [check-for-pr]
32+
name: 'Linux: Build/Test Wheels'
33+
uses: ./.github/workflows/reusable_build_and_test_wheels.yml
34+
with:
35+
PLATFORM: linux
36+
WHEEL_ARTIFACT_NAME: linux-wheel
37+
RRD_ARTIFACT_NAME: linux-rrd
38+
secrets: inherit
39+
40+
build-windows:
41+
needs: [check-for-pr]
42+
name: 'Windows: Build/Test Wheels'
43+
uses: ./.github/workflows/reusable_build_and_test_wheels.yml
44+
with:
45+
PLATFORM: windows
46+
WHEEL_ARTIFACT_NAME: windows-wheel
47+
RRD_ARTIFACT_NAME: ''
48+
secrets: inherit
49+
50+
build-macos-arm:
51+
needs: [check-for-pr]
52+
name: 'Macos-Arm: Build/Test Wheels'
53+
uses: ./.github/workflows/reusable_build_and_test_wheels.yml
54+
with:
55+
PLATFORM: macos-arm
56+
WHEEL_ARTIFACT_NAME: macos-arm-wheel
57+
RRD_ARTIFACT_NAME: ''
58+
secrets: inherit
59+
60+
build-macos-intel:
61+
needs: [check-for-pr]
62+
name: 'Macos-Intel: Build/Test Wheels'
63+
uses: ./.github/workflows/reusable_build_and_test_wheels.yml
64+
with:
65+
PLATFORM: macos-intel
66+
WHEEL_ARTIFACT_NAME: 'macos-intel-wheel'
67+
RRD_ARTIFACT_NAME: ''
68+
secrets: inherit
69+
70+
upload-wheels-linux:
71+
name: 'Linux: Upload Wheels'
72+
needs: [build-linux]
73+
uses: ./.github/workflows/reusable_upload_wheels.yml
74+
with:
75+
WHEEL_ARTIFACT_NAME: linux-wheel
76+
RRD_ARTIFACT_NAME: linux-rrd
77+
secrets: inherit
78+
79+
upload-wheels-windows:
80+
name: 'Windows: Upload Wheels'
81+
needs: [build-linux, build-windows]
82+
uses: ./.github/workflows/reusable_upload_wheels.yml
83+
with:
84+
WHEEL_ARTIFACT_NAME: windows-wheel
85+
RRD_ARTIFACT_NAME: linux-rrd
86+
secrets: inherit
87+
88+
upload-wheels-macos-arm:
89+
name: 'Macos-Arm: Upload Wheels'
90+
needs: [build-linux, build-macos-arm]
91+
uses: ./.github/workflows/reusable_upload_wheels.yml
92+
with:
93+
WHEEL_ARTIFACT_NAME: macos-arm-wheel
94+
RRD_ARTIFACT_NAME: linux-rrd
95+
secrets: inherit
96+
97+
upload-wheels-macos-intel:
98+
name: 'Macos-Intel: Upload Wheels'
99+
needs: [build-linux, build-macos-intel]
100+
uses: ./.github/workflows/reusable_upload_wheels.yml
101+
with:
102+
WHEEL_ARTIFACT_NAME: macos-intel-wheel
103+
RRD_ARTIFACT_NAME: linux-rrd
104+
secrets: inherit
105+
106+
update-pr-summary:
107+
name: 'Update PR Summary'
108+
needs: [check-for-pr, upload-wheels-linux, upload-wheels-windows, upload-wheels-macos-arm, upload-wheels-macos-intel]
109+
uses: ./.github/workflows/reusable_pr_summary.yml
110+
with:
111+
PR_NUMBER: ${{ needs.check-for-pr.outputs.PR_NUMBER}}
112+
secrets: inherit

0 commit comments

Comments
 (0)