ci: start extensive benchmark testing (execution)

.github/workflows/ci_benchmark.yml

@@ -4,10 +4,13 @@ on:
   pull_request:
   workflow_dispatch:
 
+permissions:
+  issues: write
+
 jobs:
   benchmark:
     runs-on: ubuntu-latest
-    timeout-minutes: 5
+    timeout-minutes: 15
     strategy:
       fail-fast: false
     name: Compare
@@ -48,37 +51,38 @@ jobs:
         run: npm ci
 
       - name: 🚀 Building Poku
-        run: npm run build
+        run: |
+          npm run build
+          cd benchmark && npm ci
 
       - name: 🎖️ Rock it
         run: npm run benchmark
 
-      # - name: Find Comment
-      #   uses: peter-evans/find-comment@v3
-      #   id: fc
-      #   with:
-      #     issue-number: ${{ github.event.pull_request.number }}
-      #     comment-author: 'github-actions[bot]'
-      #     body-includes: Benchmark Results
+      - name: Find Comment
+        uses: peter-evans/find-comment@v3
+        id: fc
+        with:
+          issue-number: ${{ github.event.pull_request.number }}
+          comment-author: 'github-actions[bot]'
+          body-includes: Benchmarks
 
-      # - name: 🤖 Actions - Creating Comment
-      #   if: steps.fc.outputs.comment-id == ''
-      #   uses: peter-evans/create-or-update-comment@v4
-      #   with:
-      #     issue-number: ${{ github.event.pull_request.number }}
-      #     body-path: './benchmark/output.md'
+      - name: 🤖 Actions - Creating Comment
+        if: steps.fc.outputs.comment-id == ''
+        uses: peter-evans/create-or-update-comment@v4
+        with:
+          issue-number: ${{ github.event.pull_request.number }}
+          body-path: './benchmark/output.md'
 
-      # - name: 🤖 Actions - Updating Comment
-      #   if: steps.fc.outputs.comment-id != ''
-      #   uses: peter-evans/create-or-update-comment@v4
-      #   with:
-      #     comment-id: ${{ steps.fc.outputs.comment-id }}
-      #     issue-number: ${{ github.event.pull_request.number }}
-      #     edit-mode: replace
-      #     body-path: './benchmark/output.md'
+      - name: 🤖 Actions - Updating Comment
+        if: steps.fc.outputs.comment-id != ''
+        uses: peter-evans/create-or-update-comment@v4
+        with:
+          comment-id: ${{ steps.fc.outputs.comment-id }}
+          issue-number: ${{ github.event.pull_request.number }}
+          edit-mode: replace
+          body-path: './benchmark/output.md'
 
       - name: 🐷 Ensure Poku
         run: |
           cd benchmark
-          SHORT_SHA=$(git rev-parse --short HEAD)
-          jq -e ".results | sort_by(.mean) | .[0].command == \"🐷 Poku ($SHORT_SHA)\"" results.json
+          npm run compare

.prettierignore

@@ -4,4 +4,5 @@
 /website
 /test/__fixtures__/e2e/sintax
 /.nycrc.json
-/benchmark/results.json
+/benchmark/results
+/benchmark/output.md

README.md

@@ -261,8 +261,8 @@ To see the detailed documentation, please visit the [**Documentation**](https://
 **Poku** is [continuously tested](https://github.com/wellwelwel/poku/blob/main/.github/workflows/ci_benchmark.yml) to ensure the following expectations for basic usage:
 
 - ~**4x** faster than [**Jest**](https://github.com/jestjs/jest) (v29.7.0)
-- ~**4x** faster than [**Vitest**](https://github.com/vitest-dev/vitest) (v2.1.3)
-- ~**2x** faster than [**Mocha**](https://github.com/mochajs/mocha) (v10.7.3) — _even with test file isolation_
+- ~**4x** faster than [**Vitest**](https://github.com/vitest-dev/vitest) (v3.0.6)
+- ~**1x** faster than [**Mocha**](https://github.com/mochajs/mocha) (v11.1.0) — _even with test file isolation_
 
 > - You can see how the tests are run and compared in the [benchmark](https://github.com/wellwelwel/poku/tree/main/benchmark) directory.
 > - [Comparing **Poku** and native test runners _(discussion)_](https://github.com/wellwelwel/poku/discussions/740).

benchmark/.gitignore

@@ -1,3 +1,3 @@
 node_modules
-results.json
+results
 output.md

benchmark/README.md

@@ -1,10 +1,12 @@
 # Benchmark
 
-The benchmark is performed by comparing a simple success and a failure test, each in a respective file and using parallel/concurrent runs.
-
-> [!important]
+> [!IMPORTANT]
 >
 > **Poku** doesn't intend to be "the best", but to offer a balance between high performance, isolation, compatibility, lightness and ease of use.
+>
+> - All results are based on the **Ubuntu** (`ubuntu-latest`) matrix from **GitHub Actions**.
+> - Please take into consideration that benchmarks do not indicate the competitiveness of one over the other; rather, they serve as a metric to monitor and objectively assess the current performance state of the project.
+> - To see the latest benchmarks, just check the `github-bot` comment in recent Pull Requests.
 
 ---
 
@@ -18,20 +20,44 @@ The testers to be compared are chosen based on the three most downloaded test ru
 
 **Poku** is continuously tested ([**CI**](https://github.com/wellwelwel/poku/blob/main/.github/workflows/ci_benchmark.yml)) to ensure the following expectations for basic usage:
 
+## 🏃🏻‍♀️ Execution Tests
+
+> [!NOTE]
+>
+> Focuses solely in execution, using a simple `assert(true)` or `assert(false)` from **Node.js** and searches for files in four levels of depth.
+>
+> - **success:** a suite of 5 tests that will pass.
+> - **failure:** a suite of 5 tests that will fail.
+> - **balanced:** a suite of 10 tests where 5 tests will fail and 5 tests will pass.
+
 - ~**4x** faster than [**Jest**](https://github.com/jestjs/jest) (v29.7.0)
-- ~**4x** faster than [**Vitest**](https://github.com/vitest-dev/vitest) (v2.1.3)
-- ~**2x** faster than [**Mocha**](https://github.com/mochajs/mocha) (v10.7.3) — _even with test file isolation_
+- ~**4x** faster than [**Vitest**](https://github.com/vitest-dev/vitest) (v3.0.6)
+- ~**1x** faster than [**Mocha**](https://github.com/mochajs/mocha) (v11.1.0) — _even with test file isolation_
 
 ---
 
-## Running
+More benchmarks are coming.  
+Follow the progress in [**Improve benchmarks**](https://github.com/wellwelwel/poku/issues/926) issue.
+
+---
+
+## Running Locally
 
 To run the benchmark tests, follow these steps in the `./poku` directory:
 
 ```sh
-npm ci && npm run build && npm run benchmark
-```
+# Build Poku
+npm ci
+npm run build
 
----
+# Install benchmark dependencies
+cd benchmark
+npm ci
+
+# Run and compare benchmarks
+npm start
+npm run compare
 
-- [Comparing **Poku** and native test runners _(discussion)_](https://github.com/wellwelwel/poku/discussions/740).
+# Return to Poku root
+cd ..
+```

benchmark/benchmark.sh

@@ -2,39 +2,141 @@
 
 SHORT_SHA=$(git rev-parse --short HEAD)
 
-echo '### 🚀 Benchmark Results\n'
-echo '```'
-
-hyperfine -i --warmup 5 --export-json results.json \
-  --command-name "🐷 Poku ($SHORT_SHA)" '../lib/bin/index.js ./test/poku' \
-  --command-name 'Mocha (10.7.3)' './node_modules/mocha/bin/mocha.js --parallel ./test/mocha' \
-  --command-name 'Jest (29.7.0)' 'node --experimental-vm-modules ./node_modules/jest/bin/jest.js ./test/jest' \
-  --command-name 'Vitest (2.1.3)' './node_modules/vitest/vitest.mjs run ./test/vitest' |
-  awk '/Summary/ {flag=1} flag'
-
-echo '```\n'
-echo '<details>'
-echo '<summary><h3>🔬 Native Test Runners</h3></summary>\n'
-echo '#### 🐢 Comparative with Node.js\n'
-echo '```'
-
-# Not included in results.json
-hyperfine -i --warmup 5 \
-  --command-name 'Node.js' 'node --test "./test/node/**.spec.js"' \
-  --command-name "🐷 Poku ($SHORT_SHA)" '../lib/bin/index.js ./test/poku' |
-  awk '/Summary/ {flag=1} flag'
-
-echo '```\n'
-echo '#### 🍞 Comparative with Bun\n'
-echo '```'
-
-# Not included in results.json
-hyperfine -i --warmup 5 \
-  --command-name 'Bun' 'bun test "test/bun/"' \
-  --command-name "🐷 Poku ($SHORT_SHA)" 'bun ../lib/bin/index.js ./test/poku' |
-  awk '/Summary/ {flag=1} flag'
-
-echo '```\n'
-echo '</details>\n'
-echo '> [!NOTE]'
-echo '> 📘 For more details and how the benchmarks are carried out, see the [**benchmark**](https://github.com/wellwelwel/poku/tree/main/benchmark) section.'
+HR="\n---\n"
+
+BIN_POKU="node ./node_modules/poku/lib/bin/index.js"
+BIN_MOCHA="node ./node_modules/mocha/bin/mocha.js --parallel"
+BIN_JEST="node --experimental-vm-modules ./node_modules/jest/bin/jest.js"
+BIN_VITEST="node ./node_modules/vitest/vitest.mjs run"
+BIN_NODE="node --test"
+
+rm -rf results
+
+mkdir -p results/generalist
+mkdir -p results/assertions
+mkdir -p results/execution/success
+mkdir -p results/execution/balanced
+mkdir -p results/execution/failure
+
+h1() {
+  echo "# $1\n"
+}
+
+h2() {
+  echo "## $1\n"
+}
+
+h3() {
+  echo "$HR"
+  echo "### $1"
+}
+
+quote() {
+  echo "> $1"
+}
+
+li() {
+  echo "- $1"
+}
+
+grid() {
+  quote "<details>"
+  quote "<summary>"
+  quote "See commands"
+  quote "</summary>"
+  quote "<table>"
+  quote "<tr>"
+  quote "<td>"
+  quote "source"
+  quote "</td>"
+  quote "<td>"
+  quote ""
+  quote "\`\`\`sh"
+  quote "$1"
+  quote "\`\`\`"
+  quote ""
+  quote "</td>"
+  quote "</tr>"
+  quote "<tr>"
+  quote "<td>"
+  quote "poku"
+  quote "</td>"
+  quote "<td>"
+  quote ""
+  quote "\`\`\`sh"
+  quote "$2"
+  quote "\`\`\`"
+  quote ""
+  quote "</td>"
+  quote "</tr>"
+  quote "</table>"
+  quote "</details>"
+}
+
+execution() {
+  local name=$1
+  local bin=$2
+  local dir=$3
+  local path=$4
+  local cmd_src="$bin \"./test/execution/${dir}/${path}\""
+  local cmd_poku="$BIN_POKU \"./test/execution/${dir}/poku\""
+
+  echo ""
+  li "${dir}"
+  echo ""
+  echo "\`\`\`"
+  hyperfine -i --warmup 5 --runs 10 --export-json "results/execution/${dir}/${name}.json" \
+    --command-name "$name" "$cmd_src" \
+    --command-name "🐷 Poku ($SHORT_SHA)" "$BIN_POKU ./test/execution/${dir}/poku" 2>/dev/null |
+    awk '/ ran/ {flag=1} flag'
+  echo "\`\`\`"
+  echo ""
+  grid "$cmd_src" "$cmd_poku"
+}
+
+h1 "🎖️ Benchmarks"
+
+h2 "🏃🏻‍♀️ 1/4 Execution Tests"
+
+quote "[!NOTE]"
+quote ""
+quote "Focuses solely in execution, using a simple \`assert(true)\` or \`assert(false)\` from **Node.js** and searches for files in four levels of depth."
+quote ""
+quote "- **success:** a suite of 5 tests that will pass."
+quote "- **failure:** a suite of 5 tests that will fail."
+quote "- **balanced:** a suite of 10 tests where 5 tests will fail and 5 tests will pass."
+
+echo ""
+echo "<details>"
+echo "<summary>"
+echo "See Results"
+echo "</summary>"
+
+h3 "🃏 [Jest](https://github.com/jestjs/jest)"
+execution "jest" "$BIN_JEST" "success" "jest"
+execution "jest" "$BIN_JEST" "failure" "jest"
+execution "jest" "$BIN_JEST" "balanced" "jest"
+
+h3 "⚡️ [Vitest](https://github.com/vitest-dev/vitest)"
+execution "vitest" "$BIN_VITEST" "success" "vitest"
+execution "vitest" "$BIN_VITEST" "failure" "vitest"
+execution "vitest" "$BIN_VITEST" "balanced" "vitest"
+
+h3 "☕️ [Mocha](https://github.com/mochajs/mocha)"
+execution "mocha" "$BIN_MOCHA" "success" "mocha/**"
+execution "mocha" "$BIN_MOCHA" "failure" "mocha/**"
+execution "mocha" "$BIN_MOCHA" "balanced" "mocha/**"
+
+h3 "🐢 [Node.js (built-in)](https://github.com/nodejs/node)"
+execution "node" "$BIN_NODE" "success" "node/**/**.spec.js"
+execution "node" "$BIN_NODE" "failure" "node/**/**.spec.js"
+execution "node" "$BIN_NODE" "balanced" "node/**/**.spec.js"
+
+echo ""
+echo "</details>"
+
+echo "$HR"
+
+quote "[!IMPORTANT]"
+quote ""
+quote "Please take into consideration that benchmarks do not indicate the competitiveness of one over the other; rather, they serve as a metric to monitor and objectively assess the current performance state of the project."