Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[HOLD for payment 2024-11-11] Investigate workflow job failing on main: e2ePerformanceTests / Run E2E tests in AWS device farm #49198

Closed
github-actions bot opened this issue Sep 13, 2024 · 8 comments
Closed

[HOLD for payment 2024-11-11] Investigate workflow job failing on main: e2ePerformanceTests / Run E2E tests in AWS device farm #49198

github-actions bot opened this issue Sep 13, 2024 · 8 comments
Assignees
@deetergp
Labels
Awaiting Payment Auto-added when associated PR is deployed to production Weekly KSv2 Workflow Failure

Comments

@github-actions
Copy link
Contributor

🚨 Failure Summary 🚨:

⚠️ Action Required ⚠️:

🛠️ A recent merge appears to have caused a failure in the job named e2ePerformanceTests / Run E2E tests in AWS device farm.
This issue has been automatically created and labeled with Workflow Failure for investigation.

👀 Please look into the following:

  1. Why the PR caused the job to fail?
  2. Address any underlying issues.

🐛 We appreciate your help in squashing this bug!
@saifelance
Copy link

saifelance commented Sep 14, 2024 •
edited by deetergp
Loading

1. Update Node.js Version for AWS Device Farm

Current workflow is using a deprecated version of Node.js (Node 16), you need to update the realm/aws-devicefarm action to use Node.js 20.

Update .github/workflows/e2ePerformanceTests.yml

Update the workflow file to use Node.js 20, and ensure the correct version of the realm/aws-devicefarm action is used.
Example Workflow (e2ePerformanceTests.yml):

name: e2ePerformanceTests

on:
  push:
    branches:
      - main

jobs:
  run-e2e-tests:
    runs-on: ubuntu-latest
    steps:
      # Checkout code
      - name: Checkout code
        uses: actions/checkout@v3

      # Setup Node.js version (Node.js 20)
      - name: Setup Node.js
        uses: actions/setup-node@v3
        with:
          node-version: '20'

      # Install dependencies
      - name: Install dependencies
        run: npm install

      # AWS Device Farm Configuration
      - name: Run E2E tests in AWS Device Farm
        uses: realm/aws-devicefarm@v1 # Ensure to use a compatible version
        with:
          aws_access_key_id: ${{ secrets.AWS_ACCESS_KEY_ID }}
          aws_secret_access_key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
          aws_region: 'us-west-2' # Set your region
          project_arn: ${{ secrets.AWS_DEVICEFARM_PROJECT_ARN }}
          device_pool_arn: ${{ secrets.AWS_DEVICEFARM_DEVICE_POOL_ARN }}
          app_path: './app/build/outputs/apk/debug/app-debug.apk' # Set your app path
          test_path: './app/build/outputs/test/debug/test-debug.apk'
          test_type: 'INSTRUMENTATION' # Set appropriate test type
          timeout_minutes: 120 # Adjust timeout if necessary

      # Upload logs
      - name: Upload test logs
        uses: actions/upload-artifact@v3
        with:
          name: debug.log
          path: path/to/debug.log # Ensure logs are correctly configured and uploaded

Increase Timeout and Handle Test Failures

2 . Install Required Dependencies

Installation of AWS CLI (locally or in CI):

# Install AWS CLI if not already installed
curl "https://awscli.amazonaws.com/AWSCLIV2.pkg" -o "AWSCLIV2.pkg"
sudo installer -pkg AWSCLIV2.pkg -target /

Configuring AWS CLI:

aws configure
# Enter AWS Access Key, Secret Access Key, region, and output format.

3 . Add Retry Logic in GitHub Actions:

  # Run E2E tests in AWS Device Farm with retry logic
      - name: Run E2E tests in AWS Device Farm
        uses: realm/aws-devicefarm@v1
        with:
          aws_access_key_id: ${{ secrets.AWS_ACCESS_KEY_ID }}
          aws_secret_access_key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
          aws_region: 'us-west-2'
          project_arn: ${{ secrets.AWS_DEVICEFARM_PROJECT_ARN }}
          device_pool_arn: ${{ secrets.AWS_DEVICEFARM_DEVICE_POOL_ARN }}
          app_path: './app/build/outputs/apk/debug/app-debug.apk'
          test_path: './app/build/outputs/test/debug/test-debug.apk'
          test_type: 'INSTRUMENTATION'
        continue-on-error: true # Allow the job to continue even if it fails

      # Retry failed step
      - name: Retry E2E tests if failed
        if: failure()
        run: |
          echo "Retrying failed E2E tests..."
          # Retry logic here (you can rerun specific commands if needed)

4 . Verify Node.js Version in GitHub Actions

yaml
Copy code
      # Verify Node.js version
      - name: Verify Node.js Version
        run: node -v

By following these steps, you should be able to:

  1. Fix the Node.js deprecation warning.
  2. Address the test timeout issue.
  3. Ensure that test logs are properly uploaded.
  4. Optionally, implement a retry mechanism for intermittent test failures.

Let me know if you need further clarification or help with any part of the process!

@melvin-bot melvin-bot bot added the Overdue label Sep 16, 2024
@deetergp
Copy link
Contributor

Started a convo about this in Slack.

@melvin-bot melvin-bot bot removed the Overdue label Sep 16, 2024
@deetergp deetergp added Daily KSv2 and removed Hourly KSv2 labels Sep 17, 2024
@deetergp
Copy link
Contributor

@kirillzyusko & @Szymon20000 Are investigating and sharing their results in this Slack thread.

@melvin-bot melvin-bot bot added the Overdue label Sep 20, 2024
@deetergp
Copy link
Contributor

Any news here @kirillzyusko?

@melvin-bot melvin-bot bot removed the Overdue label Sep 20, 2024
@deetergp deetergp added Weekly KSv2 and removed Daily KSv2 labels Sep 20, 2024
@kirillzyusko
Copy link
Contributor

@deetergp no, it was the only one time error, so I really don't know what caused it 🤷‍♂️ Maybe it was just a random failure 😅

Anyway, let's observe a little bit more whether we will have such issue in future or not 👀

We had this error (posting for transparency and because after some time logs will be lost):

[Fri Sep 13 2024 23:20:10 GMT+0000 (Coordinated Universal Time)]  Killing com.expensify.chat.e2e
[Fri Sep 13 2024 23:20:10 GMT+0000 (Coordinated Universal Time)]   adb shell am force-stop com.expensify.chat.e2e

[Fri Sep 13 2024 23:20:10 GMT+0000 (Coordinated Universal Time)]  ▶️ [DELTA] Test 'Report typing' [3/5], iteration [36/60]
[Fri Sep 13 2024 23:20:10 GMT+0000 (Coordinated Universal Time)]  Killing com.expensify.chat.e2edelta
[Fri Sep 13 2024 23:20:10 GMT+0000 (Coordinated Universal Time)]   adb shell am force-stop com.expensify.chat.e2edelta

[Fri Sep 13 2024 23:20:10 GMT+0000 (Coordinated Universal Time)]  Launching com.expensify.chat.e2edelta
[Fri Sep 13 2024 23:20:10 GMT+0000 (Coordinated Universal Time)]   adb shell am start -n com.expensify.chat.e2edelta/com.expensify.chat.MainActivity --ez mockNetwork true
Starting: Intent { cmp=com.expensify.chat.e2edelta/com.expensify.chat.MainActivity (has extras) }
[Fri Sep 13 2024 23:20:16 GMT+0000 (Coordinated Universal Time)]  🔙 Pressing backspace
[Fri Sep 13 2024 23:20:16 GMT+0000 (Coordinated Universal Time)]   adb shell input keyevent KEYCODE_DEL

[Fri Sep 13 2024 23:20:16 GMT+0000 (Coordinated Universal Time)]  📝 Typing text: A
[Fri Sep 13 2024 23:20:16 GMT+0000 (Coordinated Universal Time)]   adb shell input text "A"

[Fri Sep 13 2024 23:20:19 GMT+0000 (Coordinated Universal Time)]  [LISTENER] Test 'Composer typing rerender count' on 'delta' measured 1renders
[Fri Sep 13 2024 23:20:19 GMT+0000 (Coordinated Universal Time)]  🔙 Pressing backspace
[Fri Sep 13 2024 23:20:19 GMT+0000 (Coordinated Universal Time)]   adb shell input keyevent KEYCODE_DEL

[Fri Sep 13 2024 23:20:19 GMT+0000 (Coordinated Universal Time)]  📝 Typing text: Measure_performance#309046
[Fri Sep 13 2024 23:20:19 GMT+0000 (Coordinated Universal Time)]   adb shell input text "Measure_performance#309046"

[Fri Sep 13 2024 23:20:22 GMT+0000 (Coordinated Universal Time)]  [LISTENER] Test 'Message sent' on 'delta' measured 429.009887999855ms
[Fri Sep 13 2024 23:20:22 GMT+0000 (Coordinated Universal Time)]  ✅  [DELTA] Test 'Report typing' [3/5], iteration [36/60]
[Fri Sep 13 2024 23:20:22 GMT+0000 (Coordinated Universal Time)]  [LISTENER] Test 'Report typing (CPU)' on 'delta' measured 116.3481107000495%
[Fri Sep 13 2024 23:20:22 GMT+0000 (Coordinated Universal Time)]  [LISTENER] Test 'Report typing (FPS)' on 'delta' measured 59.89491953158422FPS
[Fri Sep 13 2024 23:20:22 GMT+0000 (Coordinated Universal Time)]  [LISTENER] Test 'Report typing (RAM)' on 'delta' measured 505.79811789772725MB
[Fri Sep 13 2024 23:20:22 GMT+0000 (Coordinated Universal Time)]  [LISTENER] Test 'Report typing (CPU/JS)' on 'delta' measured 59.2%
[Fri Sep 13 2024 23:20:22 GMT+0000 (Coordinated Universal Time)]  [LISTENER] Test 'Report typing (CPU/UI)' on 'delta' measured 26.5%
[Fri Sep 13 2024 23:20:22 GMT+0000 (Coordinated Universal Time)]  Killing com.expensify.chat.e2edelta
[Fri Sep 13 2024 23:20:22 GMT+0000 (Coordinated Universal Time)]   adb shell am force-stop com.expensify.chat.e2edelta

[Fri Sep 13 2024 23:20:22 GMT+0000 (Coordinated Universal Time)]  ▶️ [MAIN] Test 'Report typing' [3/5], iteration [37/60]
[Fri Sep 13 2024 23:20:22 GMT+0000 (Coordinated Universal Time)]  Killing com.expensify.chat.e2e
[Fri Sep 13 2024 23:20:22 GMT+0000 (Coordinated Universal Time)]   adb shell am force-stop com.expensify.chat.e2e

[Fri Sep 13 2024 23:20:22 GMT+0000 (Coordinated Universal Time)]  Launching com.expensify.chat.e2e
[Fri Sep 13 2024 23:20:22 GMT+0000 (Coordinated Universal Time)]   adb shell am start -n com.expensify.chat.e2e/com.expensify.chat.MainActivity --ez mockNetwork true
Starting: Intent { cmp=com.expensify.chat.e2e/com.expensify.chat.MainActivity (has extras) }
[Fri Sep 13 2024 23:25:20 GMT+0000 (Coordinated Universal Time)]  ▶️ [MAIN] Test 'Report typing' [3/5], iteration [37/60]
[Fri Sep 13 2024 23:25:20 GMT+0000 (Coordinated Universal Time)]  Killing com.expensify.chat.e2e
[Fri Sep 13 2024 23:25:20 GMT+0000 (Coordinated Universal Time)]   adb shell am force-stop com.expensify.chat.e2e
[Fri Sep 13 2024 23:25:20 GMT+0000 (Coordinated Universal Time)]  🔴  failed with error: Command failed: adb shell am force-stop com.expensify.chat.e2e
cmd: Can't find service: activity

@deetergp
Copy link
Contributor

Okay I am going to close this one out then. If it happens again, we'll know.

@kirillzyusko kirillzyusko mentioned this issue Oct 23, 2024
Merged
48 tasks
@melvin-bot melvin-bot bot added Reviewing Has a PR in review Weekly KSv2 and removed Weekly KSv2 labels Oct 23, 2024
@melvin-bot melvin-bot bot added Weekly KSv2 Awaiting Payment Auto-added when associated PR is deployed to production and removed Weekly KSv2 labels Nov 4, 2024
@melvin-bot melvin-bot bot changed the title Investigate workflow job failing on main: e2ePerformanceTests / Run E2E tests in AWS device farm [HOLD for payment 2024-11-11] Investigate workflow job failing on main: e2ePerformanceTests / Run E2E tests in AWS device farm Nov 4, 2024
@melvin-bot melvin-bot bot removed the Reviewing Has a PR in review label Nov 4, 2024
@melvin-bot Melvin Bot
Copy link

melvin-bot bot commented Nov 4, 2024

Reviewing label has been removed, please complete the "BugZero Checklist".

@melvin-bot Melvin Bot
Copy link

melvin-bot bot commented Nov 4, 2024

The solution for this issue has been 🚀 deployed to production 🚀 in version 9.0.56-9 and is now subject to a 7-day regression period 📆. Here is the list of pull requests that resolve this issue:

If no regressions arise, payment will be issued on 2024-11-11. 🎊

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@deetergp deetergp

Labels
Awaiting Payment Auto-added when associated PR is deployed to production Weekly KSv2 Workflow Failure
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@deetergp @saifelance @kirillzyusko