Skip to content

Files

Latest commit

 

History

History
292 lines (202 loc) · 11.9 KB

playwright-gitlab-ci-cd.md

File metadata and controls

292 lines (202 loc) · 11.9 KB
description
Running Playwright test in parallel using GitLab CI/CD

Playwright - GitLab CI/CD

{% hint style="info" %} TL;DR take a look at the example repository:

https://gitlab.com/currents.dev/gitlab-playwright-currents {% endhint %}

The example .gitlab-ci.yml file creates a GitLab CI pipeline with 3 workers using 3 Playwright Shards:

{% code overflow="wrap" %}

# .gitlab-ci.yml
default:
  image: mcr.microsoft.com/playwright:latest

stages:
  - test

test:
  image: mcr.microsoft.com/playwright:latest
  stage: test
  parallel: 3

  script:
    - npm ci
    - npx playwright install
    - cd ./basic
    - npx pwc --key $CURRENTS_RECORD_KEY --project-id bnsqNa --shard=$CI_NODE_INDEX/$CI_NODE_TOTAL

{% endcode %}

Running 3 parallel jobs Playwright jobs in GitLab CI

Setup

Additional resources:

GitLab CI Playwright Results

The results are being reported to Currents for more efficient troubleshooting, and monitoring test suite flakiness and performance.

Currents will collect the following information:

  • console output
  • screenshots
  • videos
  • trace files
  • timing
  • outcomes
  • flaky tests
  • error details
  • tags for more convenient management of the tests

GitLab Merge Request Notes

Take a look at the example merge request: https://gitlab.com/currents.dev/gitlab-playwright-currents/-/merge_requests/3

When GitLab integration is enabled, Currents will post a Merge Commit note with run results:

Currents Merge Request Notes

In addition, Currents will post an External Status check for every Playwright project configured. For example:

Example of GitLab CI External Status Check

Re-run only failed Playwright tests in GitLab CI/CD

When a workflow fails in GitLab CI/CD you have the option to re-run the failed jobs. However, an additional setup is required for properly configuring Playwright for rerunning only the failed tests. See Re-run Only Failed Tests guide for details.

Rerunning only the failed jobs on GitLab CI/CD

Playwright Sharding

If you're using Playwright Sharding for running your tests in parallel, use currents cache command to store the last run results and simplify re-run workflows.

Example workflows are available in our GitLab repositories:

  • reruns-pwc.yml - re-run only failed tests on GitLab CI using 3 parallel runners and Playwright Shards + Currents pwc command
  • reruns-reporter.yml - re-run only failed tests on GitLab CI using 3 parallel runners and Playwright Shards + Currents reporter in playwright.config.ts
Install the @currents/cmd package 
npm i -D @currents/cmd
Add an after_script to upload the cache

Add an after_script to the end of your job that uploads run information to the cache

after_script:
    # Save the last-run.json to cache after the run
    - npx currents cache set --pw-output-dir basic/test-results --preset last-run

See the configuration for details on the flags.

Add a download cache step to you script

Update your job script to download the cache prior to running tests

script:
    - npm ci
    # Grab the last run from cache. CacheId is automatically calculated for GitLab, and a .currents_env file is created with extra env variables
    # $EXTRA_PW_FLAGS will contain the correct --shard flag, as well as --last-failed if this was a retried job
    # $RUN_ATTEMPT will be populated with a value 1+ based on how many times this job has been retried
    - npx currents cache get --preset last-run && cat .currents_env >> $GITLAB_ENV && source .currents_env
    - npx playwright install
    # Run playwright, provide a build id with the run-attempt included and also pass $EXTRA_PW_FLAGS 
    - export CURRENTS_CI_BUILD_ID="reporter-$CI_PIPELINE_ID-$RUN_ATTEMPT"
    - npx playwright test -c playwright.config.reporter.ts $EXTRA_PW_FLAGS

See the configuration for details on the flags.

A full example

{% code lineNumbers="true" %}

test-rerun-reporter:
  image: mcr.microsoft.com/playwright:v1.47.0
  stage: test
  parallel: 3
  variables:
    CURRENTS_PROJECT_ID: bnsqNa
    # CURRENTS_RECORD_KEY: <set in your CI/CD variables>
  script:
    - npm ci
    # Grab the last run from cache. CacheId is automatically calculated for GitLab, and a .currents_env file is created with extra env variables
    # $EXTRA_PW_FLAGS will contain the correct --shard flag, as well as --last-failed if this was a retried job
    # $RUN_ATTEMPT will be populated with a value 1+ based on how many times this job has been retried
    - npx currents cache get --preset last-run && cat .currents_env >> $GITLAB_ENV && source .currents_env
    - npx playwright install
    - cd ./basic
    # Run playwright, provide a build id with the run-attempt included and also pass $EXTRA_PW_FLAGS 
    - export CURRENTS_CI_BUILD_ID="reporter-$CI_PIPELINE_ID-$RUN_ATTEMPT"
    - npx playwright test -c playwright.config.reporter.ts $EXTRA_PW_FLAGS
  after_script:
    # Save the last-run.json to cache after the run
    - npx currents cache set --pw-output-dir basic/test-results --preset last-run

{% endcode %}

Currents Orchestration

In case you're using playwright-orchestration.md for running your Playwright tests in parallel, use currents-api.md command to fetch the results of the last run from the api.


An example workflow is available in our GitLab demo repository

  • rerun-or8n.yml - rerun only failed tests on GitLab CI/CD with Currents Orchestration
Install the @currents/cmd package 
npm i -D @currents/cmd
Set CURRENTS_API_KEY environment variable

Obtain an API key (see api-keys.md) and record-key.md from Currents Dashboard and set GitLab CI/CD variable accordingly

variables:
    CURRENTS_PROJECT_ID: bnsqNa
    CURRENTS_RECORD_KEY: # set CI/CD variable
    CURRENTS_API_KEY: # set  CI/CD variable
Add an after_script to upload the cache

Add an after_script to the end of your job that uploads GitLab retry information to the cache

after_script:
    # Save the last-run.json to cache after the run
    - npx currents cache set --pw-output-dir basic/test-results --preset last-run

See currents-cache.md documentation for all the available options

Add an api get-run step to you script

Update your job script to download the cache prior to running tests, and then also grab the previous run.

{% code overflow="wrap" %}

script:
    - npm ci
    # Grab the last run from cache. CacheId is automatically calculated for GitLab, and a .currents_env file is created with extra env variables
    # $EXTRA_PWCP_FLAGS will contain the the --last-failed if this was a retried job
    # $RUN_ATTEMPT will be populated with a value 1+ based on how many times this job has been retried
    - npx currents cache get --preset last-run && cat .currents_env >> $GITLAB_ENV && source .currents_env
    # Grab the complete last run from the API so orchestration can find all the failures
    - npx currents api get-run --branch=$CI_COMMIT_REF_NAME --pw-last-run --output=basic/test-results/.last-run.json || true
    - npx playwright install
    # Run pwc-p, provide a build id with the run-attempt included, and also pass EXTRA_PWCP_FLAGS
    - npx pwc-p --ci-build-id=or8n-$CI_PIPELINE_ID-$RUN_ATTEMPT $EXTRA_PWCP_FLAGS

{% endcode %}

See currents-api.md documentation to explore all the available options

A full example

{% code lineNumbers="true" %}

test-rerun-pwcp:
  image: mcr.microsoft.com/playwright:v1.47.0
  stage: test
  parallel: 3
  variables:
    CURRENTS_PROJECT_ID: bnsqNa
    # CURRENTS_RECORD_KEY: <set in your CI/CD variables>
    # CURRENTS_API_KEY: <set in your CI/CD variables>
  script:
    - npm ci
    # Grab the last run from cache. CacheId is automatically calculated for GitLab, and a .currents_env file is created with extra env variables
    # $EXTRA_PWCP_FLAGS will contain the the --last-failed if this was a retried job
    # $RUN_ATTEMPT will be populated with a value 1+ based on how many times this job has been retried
    - npx currents cache get --preset last-run && cat .currents_env >> $GITLAB_ENV && source .currents_env
    # Grab the complete last run from the API so orchestration can find all the failures
    - npx currents api get-run --branch=$CI_COMMIT_REF_NAME --pw-last-run --output=basic/test-results/.last-run.json || true
    - npx playwright install
    # Run pwc-p, provide a build id with the run-attempt included, and also pass EXTRA_PWCP_FLAGS
    - npx pwc-p --ci-build-id=or8n-$CI_PIPELINE_ID-$RUN_ATTEMPT $EXTRA_PWCP_FLAGS
  after_script:
    # Save the last-run to cache in order to track the run attempts
    - npx currents cache set --preset last-run --pw-output-dir basic/test-results

{% endcode %}