[Spike] Investigate increased memory usage with recent Firefox versions (98+)

[tbiethman]

Current behavior

When updating our internal CI image to run with Firefox 98, one of our existing builds started timing out/crashing due to a lack of resources. Example

I was able to trace this increase back to a commit and gather some metrics before/after it was merged:

Spec ran: runner/cypress/integration/reporter.errors.spec.js
OS: macOS Monterey
System: i9 / 16GB total RAM

Commit: 0bb655e			RAM (GB)
	Run 1	Run 2	Run 3	Run 4	Run 5	Avg
Firefox 99	2.55	2.53	2.59	2.68	2.43	2.556
Firefox 99 (numTestsKeptInMemory=0)	1.39	1.18	0.953	1.32	1.29	1.2266
Chrome 101	1.55	1.69	1.74	1.59	1.6	1.634
Chrome 101 (numTestsKeptInMemory=0)	1.13	1.12	1.14	1.15	1.12	1.132

Commit: 676fc97			RAM (GB)
	Run 1	Run 2	Run 3	Run 4	Run 5	Avg	% Increase
Firefox 99	3.52	3.5	3.53	3.51	3.52	3.516	37.56
Firefox 99 (numTestsKeptInMemory=0)	3.19	3.28	3.21	3.25	3.23	3.232	163 (❗)
Chrome 101	1.63	1.58	1.66	1.59	1.59	1.61	-1.5
Chrome 101 (numTestsKeptInMemory=0)	1.27	1.28	1.26	1.27	1.26	1.268	12.01

To summarize, with 676fc97 (the origin feature branch merge) we start using a lot more memory with FF99. Chrome 101 stays relatively stable.

That spec file has 83 individual tests in it, and its snapshots can be quite large. So I also recorded runs with the numTestsKeptInMemory config value set to 0 to see what impact it had on memory usage. You can see that with 0bb655e, setting that config property had a significant impact on FF99 and cut its usage in half; however, it had a minimal effect on 676fc97. It also had a decreased effect with Chrome 101 on 676fc97, with usage up 12% from the previous commit.

We should track down where/why this memory is being held.

Desired behavior

We don't consume more memory than we have to when running with Firefox.

Test code to reproduce

1. Checkout cypress
2. Run yarn cypress:open
3. Run runner/cypress/integration/reporter.errors.spec.js in various browsers
4. Look at held memory after run completes

Cypress Version

9.6.0

Other

No response

[AtofStryker]

I was able to take a look into this. So it looks like the memory leak is coming from instances of Cypress's primaryOriginCommunicator not being cleaned up between tests, but only in runIsolatedCypress tests. This is because the isolated tests recreate the iframe in which holds the cypress instance, but does not clean up the hanging references after the instance is no longer referenced. The primary domain communicator is only created once in regular cypress tests (such as the driver suite), which is why is looks like this leak is not present for normal use cases. I believe this issue is not impacting end users for this reason, only internal tests that leverage runIsolatedCypress in the 9.x develop branch.

reporter.errors.spec.js (~3,386 MB) with primaryOriginCommunicator enabled and all methods

reporter.errors.spec.js (~2,197 MB) with primaryOriginCommunicator commented out/disabled

reporter.errors.spec.js (~3,358 MB) with primaryOriginCommunicator enabled and one event present (leak detected)

I ran this against regular cypress tests in the driver as a control group to verify my hypothesis

driver check.spec.js (~196 MB) with primaryOriginCommunicator disabled

driver check.spec.js (~194 MB) with primaryOriginCommunicator enabled and all events

Since the 10.0-release merge is imminent into develop, and the runIsolatedCypress tests are no longer run or are removed, I decided to take applicable benchmarks for cy-in-cy tests, which only execute inside chrome currently. I ran the reporter.errors.spec.ts inside the app package, which is replacing the responsibilities of the runner package.

reporter.errors.cy.ts (~702 MB) with primaryOriginCommunicator enabled and all events

reporter.errors.cy.ts (~644 MB) with primaryOriginCommunicator disabled

Since this does not seem to be a user impacting users, and fixing it is likely going to not applicable in the 10.0 version of Cypress, my recommendation is that we likely do not need to fix this, but should likely revisit this issue once firefox is supported from within a cy-in-cy context.

[AtofStryker]

To add to the control group for 10.x, memory has not looked to change much between communicator enabled/disabled

10.0 driver check.cy.js (~219.53 MB) with primaryOriginCommunicator enabled and all events

10.0 driver check.cy.js (~218.78 MB) with primaryOriginCommunicator completely disabled

[AtofStryker]

I will be adding a few clean up methods to make sure the communicator events are cleaned up in any case, just to hopefully prevent this issue in the future

[cypress-bot]

The code for this is done in cypress-io/cypress#21634, but has yet to be released.
We'll update this issue and reference the changelog when it's released.

[brian-mann]

Additionally - open mode is very different than run mode, and the debugging features of snapshots are effectively a controlled memory leak. When doing perf/memory profiling it's better to use run mode where those features are disabled.

[cypress-bot]

Released in 10.0.0.

This comment thread has been locked. If you are still experiencing this issue after upgrading to
Cypress v10.0.0, please open a new issue.