Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

improve(CI): add other platform/arch agents Build & Test blocks #453

Draft
wants to merge 78 commits into
base: main
Choose a base branch
from

Conversation

shouples
Copy link
Contributor

@shouples shouples commented Oct 24, 2024

Breaks up our main Build & Test pipeline task into platform/arch specific tasks with different agents to allow testing extension activation (and the rest of the test suite) in more than just linux amd64.

Other Make-related updates for handling dependencies and caching.

image

However

I haven't been able to get the darwin agents to cooperate at all. The linux agents have to run the tests through xvfb and the win32 agent can run the tests without any special handling, but all signs point to a lack of active GUI session for the darwin agents since we bail here:

vscode/Makefile

Lines 39 to 45 in 67f2dd8

if pgrep -x "Dock" > /dev/null; then \
echo "GUI session is active."; \
npx gulp test; \
else \
echo "No active GUI session. Aborting tests."; \
exit 1; \
fi \

Attempting to use VSCODE_CLI=1 and/or ELECTRON_RUN_AS_NODE=1 started causing runner oddities with ghost flags and deeper issues that never actually ended up running any tests.

Some related issues:

And five follow-on issues to handle the actual test failures appearing on the win32 x64 agent:

@shouples shouples added this to the 2024 Q4 General Improvements milestone Oct 24, 2024
@jlrobins jlrobins marked this pull request as ready for review November 6, 2024 21:12
@jlrobins jlrobins requested a review from a team as a code owner November 6, 2024 21:12
jlrobins
jlrobins previously approved these changes Nov 6, 2024
@shouples shouples requested a review from jlrobins November 6, 2024 21:13
@shouples shouples marked this pull request as draft November 6, 2024 21:13
@shouples shouples dismissed jlrobins’s stale review November 6, 2024 21:13

just for testing

@shouples shouples requested a review from a team November 6, 2024 21:13
@shouples shouples changed the title improve(CI): add other platform/arch agents improve(CI): add other platform/arch agents Build & Test blocks Nov 15, 2024
@shouples shouples self-assigned this Nov 18, 2024
@airlock-confluentinc airlock-confluentinc bot force-pushed the djs/update-ci-for-multiple-agents branch from a232e08 to 88a8cb0 Compare January 15, 2025 14:52
@airlock-confluentinc airlock-confluentinc bot force-pushed the djs/update-ci-for-multiple-agents branch 3 times, most recently from 183211e to d5d7fcf Compare March 15, 2025 15:50
Comment on lines -22 to -37
global_job_config:
prologue:
commands:
- checkout
- make ci-bin-sem-cache-restore
- |
if [[ "${SEMAPHORE_ORGANIZATION_URL}" == *".semaphoreci.com" ]]; then
echo "Skipping Vault setup for Semaphore CI"
else
. vault-setup
fi
epilogue:
always:
commands:
- make ci-bin-sem-cache-store
- make store-test-results-to-semaphore
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Mainly pushed into Build & Test (VS Code: linux x64) with YAML anchors for reuse on the non-Windows tasks

Comment on lines 59 to 68
- name: "Build & Test (VS Code: linux arm64)"
dependencies: []
skip:
when: "branch != 'main' AND change_in(['/release.svg', '/.versions/next.txt'], {pipeline_file: 'ignore', branch_range: '$SEMAPHORE_GIT_COMMIT_RANGE', default_branch: 'main'})"
skip: *build-test-skip-stable
task:
agent:
machine:
type: s1-prod-ubuntu24-04-arm64-1
prologue: *build-test-prologue
jobs: *build-test-jobs-stable
epilogue: *build-test-epilogue
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Blocks basically copy the Build & Test (VS Code: linux x64) block apart from their agent.machine.type

commands:
- npm ci --prefer-offline --include=dev
- npx playwright install
- npx gulp build
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We can't run gulp ci here because lint will fail until #1246 is merged, and check will throw ~11 TS errors (even though we get zero TS errors on non-Windows agents).

Comment on lines +39 to +46
if pgrep -x "Dock" > /dev/null; then \
echo "GUI session is active."; \
npx gulp test; \
else \
echo "No active GUI session. Aborting tests."; \
exit 1; \
fi \
Copy link
Contributor Author

@shouples shouples Mar 15, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Darwin agents shouldn't even be re-attempted until we know we'll be able to get past this check.

Comment on lines +110 to +117
if [ $$(stat -f%z $(EXECUTABLE_DOWNLOAD_PATH) 2>/dev/null || stat -c%s $(EXECUTABLE_DOWNLOAD_PATH)) -lt 1048576 ]; then \
echo "Error: Downloaded sidecar executable is too small (< 1MB), likely corrupted or failed download" >&2; \
cat $(EXECUTABLE_DOWNLOAD_PATH) | head -20 >&2; \
rm -f $(EXECUTABLE_DOWNLOAD_PATH); \
exit 1; \
fi && \
echo "Downloaded sidecar executable to $(EXECUTABLE_DOWNLOAD_PATH) ($$(stat -f%z $(EXECUTABLE_DOWNLOAD_PATH) 2>/dev/null || stat -c%s $(EXECUTABLE_DOWNLOAD_PATH)) bytes)";
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Extra check in case we get a bad curl response and end up with <1kb text file of error response content

@airlock-confluentinc airlock-confluentinc bot force-pushed the djs/update-ci-for-multiple-agents branch from 75e6c3c to cfd1897 Compare March 16, 2025 00:02
@shouples shouples marked this pull request as ready for review March 16, 2025 00:37
@shouples shouples requested a review from rohitsanj March 16, 2025 14:29
Comment on lines -629 to +630
"--disable-gpu-sandbox",
"--disable-gpu",
Copy link
Contributor Author

@shouples shouples Mar 16, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No longer a valid flag based on code --help on the latest stable version:

❯ code --help
Visual Studio Code 1.98.2

Usage: code [options][paths...]

To read from stdin, append '-' (e.g. 'ps aux | grep code | code -')

Options
  -d --diff <file> <file>                    Compare two files with each other.
  -m --merge <path1> <path2> <base> <result> Perform a three-way merge by providing paths for two modified versions of a
                                             file, the common origin of both modified versions and the output file to save
                                             merge results.
  -a --add <folder>                          Add folder(s) to the last active window.
  --remove <folder>                          Remove folder(s) from the last active window.
  -g --goto <file:line[:character]>          Open a file at the path on the specified line and character position.
  -n --new-window                            Force to open a new window.
  -r --reuse-window                          Force to open a file or folder in an already opened window.
  -w --wait                                  Wait for the files to be closed before returning.
  --locale <locale>                          The locale to use (e.g. en-US or zh-TW).
  --user-data-dir <dir>                      Specifies the directory that user data is kept in. Can be used to open
                                             multiple distinct instances of Code.
  --profile <profileName>                    Opens the provided folder or workspace with the given profile and associates
                                             the profile with the workspace. If the profile does not exist, a new empty
                                             one is created.
  -h --help                                  Print usage.

Extensions Management
  --extensions-dir <dir>              Set the root path for extensions.
  --list-extensions                   List the installed extensions.
  --show-versions                     Show versions of installed extensions, when using --list-extensions.
  --category <category>               Filters installed extensions by provided category, when using --list-extensions.
  --install-extension <ext-id | path> Installs or updates an extension. The argument is either an extension id or a path to
                                      a VSIX. The identifier of an extension is '${publisher}.${name}'. Use '--force'
                                      argument to update to latest version. To install a specific version provide
                                      '@${version}'. For example: 'vscode.csharp@1.2.3'.
  --pre-release                       Installs the pre-release version of the extension, when using --install-extension
  --uninstall-extension <ext-id>      Uninstalls an extension.
  --update-extensions                 Update the installed extensions.
  --enable-proposed-api <ext-id>      Enables proposed API features for extensions. Can receive one or more extension IDs
                                      to enable individually.

Troubleshooting
  -v --version                    Print version.
  --verbose                       Print verbose output (implies --wait).
  --log <level>                   Log level to use. Default is 'info'. Allowed values are 'critical', 'error', 'warn',
                                  'info', 'debug', 'trace', 'off'. You can also configure the log level of an extension by
                                  passing extension id and log level in the following format:
                                  '${publisher}.${name}:${logLevel}'. For example: 'vscode.csharp:trace'. Can receive one
                                  or more such entries.
  -s --status                     Print process usage and diagnostics information.
  --prof-startup                  Run CPU profiler during startup.
  --disable-extensions            Disable all installed extensions. This option is not persisted and is effective only when
                                  the command opens a new window.
  --disable-extension <ext-id>    Disable the provided extension. This option is not persisted and is effective only when
                                  the command opens a new window.
  --sync <on | off>               Turn sync on or off.
  --inspect-extensions <port>     Allow debugging and profiling of extensions. Check the developer tools for the connection
                                  URI.
  --inspect-brk-extensions <port> Allow debugging and profiling of extensions with the extension host being paused after
                                  start. Check the developer tools for the connection URI.
  --disable-lcd-text              Disable LCD font rendering.
  --disable-gpu                   Disable GPU hardware acceleration.
  --disable-chromium-sandbox      Use this option only when there is requirement to launch the application as sudo user on
                                  Linux or when running as an elevated user in an applocker environment on Windows.
  --telemetry                     Shows all telemetry events which VS code collects.

Subcommands
  tunnel       Make the current machine accessible from vscode.dev or other machines through a secure tunnel
  serve-web    Run a server that displays the editor UI in browsers.

@airlock-confluentinc airlock-confluentinc bot force-pushed the djs/update-ci-for-multiple-agents branch from cfd1897 to 8ac117c Compare March 17, 2025 13:31
Copy link
Contributor

@rohitsanj rohitsanj left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, @shouples! I have a few minor comments.

Also, it looks like there are a few tests failing, do we need to fix those?

@airlock-confluentinc airlock-confluentinc bot force-pushed the djs/update-ci-for-multiple-agents branch from 3849e91 to 2614741 Compare March 18, 2025 15:40
…th (#1268)

* handle weirdly stringified Uri case for windows

* update test for easier maintenance
@shouples
Copy link
Contributor Author

@rohitsanj when I run this branch on my Windows machine (with the various Windows test fixes), it's passing for me with the occasional test flake, but it looks like running in CI is much more flaky in general and the failures the last couple runs have been mixed. I could definitely see these eventually turning into their own pipelines, but for now I think this does what we originally wanted.

@shouples shouples marked this pull request as draft March 20, 2025 13:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment