Get QA Auth Smoke Tests passing

[coverbeck]

Description
Get rid of the --record option when running auth tests against QA. The big con is that if the tests fail, we'll have to manually run them locally to see what the problem is, as the --record option lets us save videos where we can see the failures.

I spent about 8 hours trying to get the tests to pass on CircleCI and this is all I could figure out to get them passing. Here's what I tried, none of which worked; if anybody else has any other ideas, I'm happy to give them a shot first.

• Upped the CircleCI resource_class from medium (the default) to both medium+ and large.
• Tried a newer Docker image, that went from Node 16 to Node 18 (as well as updated the Java version). Not only did it not work, it also broke the regular integration tests. I didn't bother looking into the regular failures since it didn't fix the smoke tests anyway.
• Tried a different testing reporter, teamcity instead of junit.
• Tried a new cache
• Tried restructuring the auth-tests.ts file.

Some additional observations:

• Auth tests run locally against QA with the record option, although I'm on a Mac/ARM, so it's not an exact comparison.
• The non-auth tests run on CircleCI against QA with record with no problems.

Review Instructions
Nightly auth tests should be passing on CircleCI.

Issue
SEAB-6459

Security
If there are any concerns that require extra attention from the security team, highlight them here.

Please make sure that you've checked the following before submitting your pull request. Thanks!

• Check that your code compiles by running npm run build
• Ensure that the PR targets the correct branch. Check the milestone or fix version of the ticket.
• If this is the first time you're submitting a PR or even if you just need a refresher, consider reviewing our style guide
• Do not bypass Angular sanitization (bypassSecurityTrustHtml, etc.), or justify why you need to do so
• If displaying markdown, use the markdown-wrapper component, which does extra sanitization
• Do not use cookies, although this may change in the future
• Run npm audit and ensure you are not introducing new vulnerabilities
• Do due diligence on new 3rd party libraries, checking for CVEs
• Don't allow user-uploaded images to be served from the Dockstore domain
• If this PR is for a user-facing feature, create and link a documentation ticket for this feature (usually in the same milestone as the linked issue). Style points if you create a documentation PR directly and link that instead.
• Check whether this PR disables tests. If it legitimately needs to disable a test, create a new ticket to re-enable it in a specific milestone.

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 40.26%. Comparing base (6a4ee28) to head (d9ac54e).

Additional details and impacted files

@@           Coverage Diff            @@
##           develop    #1975   +/-   ##
========================================
  Coverage    40.26%   40.26%           
========================================
  Files          400      400           
  Lines        11992    11992           
  Branches      2901     2901           
========================================
  Hits          4829     4829           
  Misses        4872     4872           
  Partials      2291     2291           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[denis-yuen]

I don't remember the backstory on this.

Why do only the test-local-no-auth, test-qa-no-auth and test-qa-auth jobs have --record in the first place. i.e. we don't record in staging or prod

Overall, not objecting, but how did you come to suspect this as a potential fix and is there a theory for why? Or why it continues to work for those other jobs?

[coverbeck]

I don't remember the backstory on this.

Why do only the test-local-no-auth, test-qa-no-auth and test-qa-auth jobs have --record in the first place. i.e. we don't record in staging or prod

I don't know the background, but my guess is because QA is changing daily, the possibility for breakage is higher, and it's more convenient to have the artifacts so you can debug. Whereas once the tests are passing in staging/prod, it'll be months before the possibility of them breaking again happens (unless they tests themselves change).

For test-local-no-auth, my guess is so that we could test the Cypress Cloud/record integration.

Overall, not objecting, but how did you come to suspect this as a potential fix and is there a theory for why? Or why it continues to work for those other jobs?

I looked at what's different between the passing and failing jobs and the --record was a thing. I was surprised it made a difference, especially given that test-qa-no-auth works with the switch.

After that worked, I looked around, and there were some bug reports/comments of --record being an issue. Here (old ticket but still) and here.

Also, FWIW, I noted this in the JIRA issue but not in the PR, these failures started when Cypress 13 was merged to develop.

In any case, my theory is there is some flakiness in Cypress. :)

[kathy-t]

Does staging-auth work with recording?

Also, this probably won't make a difference but I noticed that there's a new browser-tools version so could try updating that to 1.4.8

dockstore-ui2/.circleci/config.yml

Line 6 in 6a4ee28

browser-tools: circleci/browser-tools@1.4.7

[coverbeck]

Does staging-auth work with recording?

Good idea, that would see if the problem stems from something new in Dockstore 1.16 or if it's purely Cypress.

Also, this probably won't make a difference but I noticed that there's a new browser-tools version so could try updating that to 1.4.8

dockstore-ui2/.circleci/config.yml

Line 6 in 6a4ee28

browser-tools: circleci/browser-tools@1.4.7

I think I tried that and it didn't make a difference. But I'll try again to be sure, and we might as well update anyway.

[svonworl]

"Get rid of the --record option when running auth tests against QA"

Recording the tests was causing them to fail? If yes, is there clues/evidence as to why that's happening?

[svonworl]

"Get rid of the --record option when running auth tests against QA"

Recording the tests was causing them to fail? If yes, is there clues/evidence as to why that's happening?

Ooops, nevermind. missed some of the comments, they discuss it.

[coverbeck]

Does staging-auth work with recording?

It also failed against staging with recording.

[coverbeck]

Also, this probably won't make a difference but I noticed that there's a new browser-tools version so could try updating that to 1.4.8

Also didn't make a difference, alas. https://app.circleci.com/pipelines/github/dockstore/dockstore-ui2/11963/workflows/657f017b-212a-4f14-98d9-5a6ce40b88e1/jobs/53876

[kathy-t]

Thanks for testing it!

[sonarqubecloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
No data about Coverage
No data about Duplication

See analysis details on SonarCloud