Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(video): add audits recording litemode option #47

Merged
merged 2 commits into from
Sep 22, 2023
Merged

feat(video): add audits recording litemode option #47

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
a486c85
feat(video): add audits recording litemode option
j-mendez Sep 22, 2023
d09958c
chore(cli): fix recording env
j-mendez Sep 22, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion .github/workflows/action.yml
View file
Original file line number Diff line number Diff line change
@@ -10,11 +10,12 @@ jobs:
uses: ./
with:
WEBSITE_URL: https://a11ywatch.com
FAIL_TOTAL_COUNT: 10
FAIL_TOTAL_COUNT: 100
EXTERNAL: false
SITE_WIDE: true
SITEMAP: true
SUBDOMAINS: true
TLD: true
LIST: true
UPGRADE: true
RECORD: recordings
55 changes: 7 additions & 48 deletions README.md
View file
Original file line number Diff line number Diff line change
@@ -21,6 +21,7 @@ When running locally the action uses A11yWatch Lite.
LIST: true
FIX: false
UPGRADE: false
RECORD: ./recordings
COMPUTER_VISION_SUBSCRIPTION_KEY: ${{ secrets.COMPUTER_VISION_SUBSCRIPTION_KEY }}
COMPUTER_VISION_ENDPOINT: ${{ secrets.COMPUTER_VISION_ENDPOINT }}
```
@@ -38,6 +39,7 @@ All inputs are **optional** except $WEBSITE_URL.
| `SITEMAP` | Extend crawl with sitemap links (required SITE_WIDE=true). | true |
| `TLD` | Include all tld extensions (required SITE_WIDE=true). | true |
| `LIST` | Report the results to github as a pass or fail list or detailed report. | false |
| `RECORD` | Record the audit as video to a directory. | |
| `FAIL_TOTAL_COUNT` | Determine whether to fail the CI if total issues warnings and errors exceed the counter. Takes precedence over the other FAIL inputs. | 0 |
| `FAIL_ERRORS_COUNT` | Determine whether to fail the CI if total issues with errors exceed the counter. | 0 |
| `FAIL_WARNINGS_COUNT` | Determine whether to fail the CI if total issues with warnings exceed the counter. | 0 |
@@ -56,61 +58,18 @@ All inputs are **optional** except $WEBSITE_URL.
| -------- | -------------------------- | ------- |
| `issues` | The amount of issues found | |

## Benches

The hardware specs for results:
## Performance

```sh
----------------------
linux ubuntu-latest
2-core CPU
7 GB of RAM memory
14 GB of SSD disk space
-----------------------
```

### Benchmark Results

The results for crawling a website that is small - large.

#### crawl-speed

Test url: `https://a11ywatch.com`

25 pages

runs with 10 samples:

| | `libraries` |
| :--------------------- | :--------------------- |
| **`A11yWatch: crawl`** | `0.4 s` (✅ **1.00x**) |
| **`Pa11y-CI: crawl`** | `45 s` (✅ **1.00x**) |
| **`Axe: crawl`** | `N/A` (✅ **1.00x**) |

#### crawl-speed (Large Website)

Test url: `https://www.hbo.com`

7500 pages.

runs with 10 samples:

| | `libraries` |
| :--------------------- | :------------------------ |
| **`A11yWatch: crawl`** | `2.5 mins` (✅ **1.00x**) |
| **`Pa11y-CI: crawl`** | `50+ hr` (✅ **1.00x**) |
| **`Axe: crawl`** | `N/A` (✅ **1.00x**) |

### Benchmark Info

On a larger website A11yWatch action runs over 60x-10,000x+ faster depending on CPUs/hardware.
On a larger website A11yWatch action runs over 60x-10,000x+ faster depending on CPUs/hardware. After the first installation you should have faster setup time between runs.
You can expect to handle at least 1k pages per minute on a 2-core CPU 7 GB of RAM memory shared github action. If you enable `RECORD` the output time may increase a bit.

When `AI_DISABLED` is set to true the run for `A11yWatch` may increase.

## Common Issues

If you experience issues on your CI you may have to toggle the `UPGRADE` input to true in order to get the latest docker images.
We need docker in order to build the appliciation in quickly since we have some services that need to compile that may take awhile.
If you experience issues on your CI you may have to toggle the `UPGRADE` input to true in order to get the latest installs. If you see
a playwright error try adding `PLAYWRIGHT_VERSION` env variable with the newest version to install chrome.

## CLI

50 changes: 47 additions & 3 deletions action.yml
View file
Original file line number Diff line number Diff line change
@@ -8,7 +8,7 @@ author: A11yWatch
branding:
icon: "shield"
color: "gray-dark"
description: Powerful web accessibility testing.
description: Fast and feature-packed web accessibility testing.
inputs:
WEBSITE_URL:
description: Website domain to scan
@@ -37,6 +37,10 @@ inputs:
description: Include all TLD extensions.
required: false
default: false
RECORD:
description: Enable video recording audits to a directory location.
required: false
default: ""
FAIL_TOTAL_COUNT:
description: Use to fail the CI if the amount of issues of type errors and warnings is greater than the count.
required: false
@@ -85,6 +89,9 @@ outputs:
issues:
description: "The amount of issues found on the page"
value: ${{ steps.a11ywatch-results-generator.outputs.issues }}
recording_dir:
description: "The directory that holds recordings of audits. Only enabled when inputs.RECORD is set."
value: ${{ steps.a11ywatch-recordings-directory.outputs.recording_dir }}
runs:
using: composite
steps:
@@ -108,6 +115,15 @@ runs:
restore-keys: |
${{ runner.os }}-yarn-

- name: Cache playwright binaries
if: ${{ inputs.EXTERNAL == 'false' }}
uses: actions/cache@v3
id: playwright-cache
with:
path: |
~/.cache/ms-playwright
key: ${{ runner.os }}-playwright-${{ env.PLAYWRIGHT_VERSION }}

- name: Cache cargo crates
uses: actions/cache@v3
id: cache-cargo
@@ -135,15 +151,15 @@ runs:
uses: baptiste0928/cargo-install@v1
with:
crate: a11ywatch_cli
version: "0.10.4"
version: "0.10.8"
features: "grpc"

- name: A11yWatch CLI Install
if: ${{ inputs.SLIM == 'false' }}
uses: baptiste0928/cargo-install@v1
with:
crate: a11ywatch_cli
version: "0.10.4"
version: "0.10.8"

- name: Configure CLI creds
if: ${{ inputs.A11YWATCH_TOKEN }}
@@ -160,11 +176,27 @@ runs:
shell: bash
run: a11ywatch --set-cv-url ${{ inputs.COMPUTER_VISION_ENDPOINT }}

- name: Configure Recording
if: ${{ inputs.RECORD }}
shell: bash
run: a11ywatch --set-recording ${{ inputs.RECORD }}

- uses: FedericoCarboni/setup-ffmpeg@v2
if: ${{ inputs.RECORD && inputs.RECORD != 'false' }}

- name: Build A11yWatch Service
if: ${{ inputs.EXTERNAL == 'false' }}
shell: bash
run: a11ywatch build --standalone --local ${{ inputs.UPGRADE == 'true' && '--upgrade' || '' }}

- run: npx playwright install --with-deps
if: ${{ inputs.EXTERNAL == 'false' && steps.playwright-cache.outputs.cache-hit != 'true' }}
shell: bash

- run: npx playwright install-deps
if: ${{ inputs.EXTERNAL == 'false' && steps.playwright-cache.outputs.cache-hit != 'true' }}
shell: bash

- name: Start A11yWatch Service
if: ${{ inputs.EXTERNAL == 'false' }}
shell: bash
@@ -273,6 +305,18 @@ runs:
message: "fix(accessibility): add recommendations from scan"
committer_name: GitHub Actions

- name: Get recording directory
id: a11ywatch-recordings-directory
if: ${{ inputs.RECORD && inputs.RECORD != 'false' && failure() || success() }}
shell: bash
run: echo "recording_dir=$( echo "$(a11ywatch --find-tmp-dir)""$inputs.RECORD" )" >> $GITHUB_OUTPUT

- uses: actions/upload-artifact@v3
if: ${{ inputs.RECORD && inputs.RECORD != 'false' && failure() || success() }}
with:
name: Recordings
path: ${{ steps.a11ywatch-recordings-directory.outputs.recording_dir }}

- name: Stop A11yWatch Service
if: ${{ !inputs.EXTERNAL }}
shell: bash