Out of memory issue after update a browsers to version 104+ (Chrome&Edge)

[MusolQA]

Current behavior

We have configured build to run UI tests on both Edge and Chrome browsers.
Last time I noticed that on Edge test are not executed and finished after 6h (pipeline timeout).
Now we have the same situation on Chrome, that test are not executed.
Without any error/log etc.

What interesting both starting to timeout after update Edge from version 103 to 104 and Chrome from version 103 to 104.
No any code changes between update from 103 to 104, so same code base failing after update to 104.

I run it locally and always have "Out of memory" issue screen during execution. --headed flag added to investigate
I try it on different versions, and always have same issue.

Desired behavior

Same code should working on browsers version 103 and 104.

Test code to reproduce

No any code, test are not running, because of out of memory issue.

Cypress Version

9.3.1, 9.7.0, 10.0

Node version

16.16.0

Operating System

Windows Azure Virtual Machine

Debug Logs

No response

Other

No response

[jsflint007]

Also seeing the same issue, no major code changes (nothing that would cause any sort of memory leak) and only the updated versioning of browsers. Debugging was always an issue for us with memory crashes but now we are seeing every agent failing to run a test in our azure pipelines due to what we suspect now is a memory issue with the chrome browser being used. Switched to Electron in the same pipeline and we had no issues

[MusolQA]

I haven't issues with electron so far. Till update Electron to version 94. After that same issue occurs for all browsers. Super weird.

[vaibhavganatra]

I'm having the same issue with chrome 104 and chrome 105 beta, edge and other latest chromium versions.

[enriquejortizg]

I'm having the same issue. What I try to do at the moment is to atomize as much as possible each spec, by making spec change it seems that the memory is "reset" and the problem does not occur, but even if there is only one test case and it is too long, the problem will occur. The most tedious thing is that it doesn't jump any error, it just freezes.

[astraljames]

Following up with a conversation on the Cypress Discord server,
I'm running Chrome 104 with Cypress 9.5.4, running headed, and I'm not having this issue, though others definitely are. The folks who are mentioned that one of them was running 9.6.1, which was receiving the OOM behavior.

My longest spec is 50+ test cases, some of which are individually 50+ lines long for reference, in case spec size is a factor.

If there's any further info I can provide please let me know.

[richardszanyi-kasa]

Same issue, chrome 104, same with electron, same like previous versions of cypress or browsers:

• long scenarios takes up the browser's memory and crashes.
• Cypress 9.6 was the same
• Now running Cypress 10.6.0 - same

Tried every option available in github issues such as:

1. setting chrome memory allocation unsuccessfully
2. numTestsKeepInMemory: 1 - no use

It would be nice if we could config chrome at least to take up more memory from the computer. :(

PS: Headless is not affected, only headed runs.

[rachelruderman]

Hi @MusolQA ! Thank you for reporting this issue, I'm sorry to hear you and others are running into this problem. In order to investigate, we'll need to be able to reproduce on our end. Can you set up a cypress-tiny-test that reproduces the issue? Thanks

[MusolQA]

@rachelruderman

Thanks for your answer.
https://github.com/MusolQA/cypress-test-tiny

Here repository with code that causes Out of memory issue.
Please install dependencies, I reproduced in on my local Windows10 machine.

To run please install dependencies (preprocessor to run BDD), and run test.feature file. Below my results.

I add a bunch of When, Then steps in feature file, but basically its failing after ~2-3 mins of each scenario execution.

Headless mode from CMD

Open via Cypress GUI

[MusolQA]

Btw. I can see that similar issue was reported and solved in January, here: #17853
Seems that its back, maybe it help you where looking for a problem.
I think it should be major priority, cause we are stuck due to issue.

[vaibhavganatra]

Btw. I can see that similar issue was reported and solved in January, here: #17853 Seems that its back, maybe it help you where looking for a problem. I think it should be major priority, cause we are stuck due to issue.

@MusolQA That was reported by me, and it's not the same issue. Issue#17853 was caused due to changes in cypress 8.2 and above while this new issue is caused due to changes in memory usage for Chrome 104 and 105 beta.
I do however agree that it should be considered a high priority since we are having to run tests on chrome 103 and aren't able to run tests on chrome 104 and 105 beta versions

[robrich7]

it is due to the new browser versions. I didn't change anything in my code but after the browser update to Chrome 104, many test specs crashed. I downgraded to Chrome 102 and the problems stopped.

[MusolQA]

Yes, definitely problem is between cypress and latest chromium relation. However you know, downgrade the browsers version might be workaround for few days, but tests should be run on latest version to be reliable.

[alecmestroni]

Same issue here

[vaibhavganatra]

See one of the several issues that have been reported on google support for chrome 104 memory.
https://support.google.com/chrome/thread/174889666/increasing-the-memory-limit-on-chrome-v104?hl=en

[PawelRuszlewicz]

@rachelruderman any update or progress?

[robrich7]

@rachelruderman any update or progress?

i don't think it's Cypress, but a problem with the Chrome browser, as you can see one post above.

[PawelRuszlewicz]

And it's final answer? Due to this chrome change and limitation we cannot run tests with take more 2mins on Chrome/Edge?
As a workaround we can use Firefox that doesn't have this issue with memory

[enriquejortizg]

At least Cypress should fail the test case when the memory problem occurs. At this point it just hangs.

[JessefSpecialisterren]

Potentially relevant Chromium issue: https://crbug.com/1353749

Edit: and another one: https://crbug.com/1352649

[rachelruderman]

Hi @MusolQA , thank you for creating the cypress-tiny-test repro! I ran it on my personal machine (Mac OS) with Chrome 104 and it passed. Can you tell me a bit more about your Windows Azure Virtual Machine? What OS version are you on and how much memory does it have? TIA!

[MusolQA]

@rachelruderman , Thanks.
Super weird that on MacOS its working fine...

Below our Azure VM configuration:

• OS
  Microsoft Windows Server 2019
  10.0.17763
  Image: windows-2019
  Version: 20220821.1
  https://github.com/actions/runner-images/blob/win19/20220821.1/images/win/Windows2019-Readme.md
• RAM
  t2.2xlarge - which equates to 32G RAM

[PawelRuszlewicz]

Also I've same issue on local machine with Win10 Pro:

[PawelRuszlewicz]

Update: After update to Chrome 105, same issue.

[kpturner]

@JessefSpecialisterren wrote

Therefore, we switched back to Chromium 99 to ensure maximum CI stability

This is becoming a bit of a nightmare. Up until version 12 I could at least get our tests to run through by forcing Cypress to use Chrome 99. With version 12 not even that browser works. I cannot get our tests to complete without OOM errors in any browser at all......which kinda rules V12 out for us for the foreseeable.

[nickpm21]

So as per previous conversations it looks like while using the GUI via "cypress open", on any browser, Chrome, Firefox, Edge, etc.. The memory used by the browser instance does not go down after finishing a test. Because this happens on multiple browsers all the same way this would seem to be pretty isolated to Cypress itself causing the issue as opposed to a Chromium version memory leak issue. Is there any way to reset memory usage for the instance while having it active? @mschile

[JessefSpecialisterren]

@kpturner wrote

This is becoming a bit of a nightmare. Up until version 12 I could at least get our tests to run through by forcing Cypress to use Chrome 99. With version 12 not even that browser works. I cannot get our tests to complete without OOM errors in any browser at all......which kinda rules V12 out for us for the foreseeable.

Interesting! We switched to Cypress 12 last week and haven't had a single OOM problem in our CI runs with Chromium 99 so far. This issue seems to be remarkably hard to pin down 🤔

For the record: does "any browser at all" include Firefox?

[henricook]

It's only newer browsers, or at least is only particularly prominent in newer browsers. We've had to keep our CI browser tagged at around Chromium 102 to keep our tests running while the Cypress team try and figure out this problem.

[kpturner]

@kpturner wrote

This is becoming a bit of a nightmare. Up until version 12 I could at least get our tests to run through by forcing Cypress to use Chrome 99. With version 12 not even that browser works. I cannot get our tests to complete without OOM errors in any browser at all......which kinda rules V12 out for us for the foreseeable.

Interesting! We switched to Cypress 12 last week and haven't had a single OOM problem in our CI runs with Chromium 99 so far. This issue seems to be remarkably hard to pin down 🤔

For the record: does "any browser at all" include Firefox?

Yes it includes Firefox but I have not tried all different versions of Firefox. It’s just not practical to try every version of every browser so I can’t say all versions of FF fail in the same way.

I’m considering splitting our test suite in two as some sort of workaround - but this situation is not good.

Firefox fails in a different way: The automation client disconnected. Cannot continue running tests. (we are not using Docker).

[kpturner]

It's only newer browsers, or at least is only particularly prominent in newer browsers. We've had to keep our CI browser tagged at around Chromium 102 to keep our tests running while the Cypress team try and figure out this problem.

If only that were true

[mschile]

@kpturner, could you please log a new issue for Firefox since this one pertains to Chromium browsers.

You can try forcing garbage collection in Chromium based browsers by adding the following in your support file:

// force garbage collection before every 5th test
let testCount = 0
beforeEach(() => {
  testCount++
  if (Cypress.isBrowser({ family: 'chromium' }) && testCount % 5 === 0) {
    Cypress.automation('remote:debugger:protocol', { command: 'HeapProfiler.collectGarbage' })
  }
})

[kpturner]

@kpturner, could you please log a new issue for Firefox since this one pertains to Chromium browsers.

You can try forcing garbage collection in Chromium based browsers by adding the following in your support file:

// force garbage collection before every 5th test
let testCount = 0
beforeEach(() => {
  testCount++
  if (Cypress.isBrowser({ family: 'chromium' }) && testCount % 5 === 0) {
    Cypress.automation('remote:debugger:protocol', { command: 'HeapProfiler.collectGarbage' })
  }
})

@mschile I will give that a try. However I have tests that crash chrome (using the GUI) after just two tests have run. I am almost at the point of trying this for every test - no idea how much it will impact performance though.

My Firefox comment was just to answer the question posed to me (had I tried it). I will log an issue later but I don't want to do so without all the baggage that goes with doing so. At the moment I need to firefight the Chrome problem as a priority.

[mschile]

@kpturner, if you are getting OOM after a few tests in open (GUI) mode, I would also suggest setting numTestsKeptInMemory: 0 in your Cypress config. This will prevent DOM snapshots from getting saved along with some other objects. This can be helpful for long tests or for a more memory intensive site.

[kpturner]

@kpturner, if you are getting OOM after a few tests in open (GUI) mode, I would also suggest setting numTestsKeptInMemory: 0 in your Cypress config. This will prevent DOM snapshots from getting saved along with some other objects. This can be helpful for long tests or for a more memory intensive site.

We have always used that setting because we have always had issues. Unfortunately with V12 it makes no difference. The chrome browser crashes every time we run certain tests.

[kpturner]

@mschile

@kpturner, could you please log a new issue for Firefox since this one pertains to Chromium browsers.

You can try forcing garbage collection in Chromium based browsers by adding the following in your support file:

// force garbage collection before every 5th test
let testCount = 0
beforeEach(() => {
  testCount++
  if (Cypress.isBrowser({ family: 'chromium' }) && testCount % 5 === 0) {
    Cypress.automation('remote:debugger:protocol', { command: 'HeapProfiler.collectGarbage' })
  }
})

Tried this overnight. Unfortunately same result

[kpturner]

Well after much pain and trial and error it turns out that I was able to return our tests (on Chrome 99 at least) back to normal simply by setting e2e.testIsolation to false in the config. Sure I had to fix a couple of breaking changes to do with aliases but no OOM errors and everything runs cleanly. I still need to try later versions of Chrome but I suspect they will still fail.

[kpturner]

So, rather astonishingly, our OOM issues have been completely tipped on their head simply by setting e2e.testIsolation to false. Prior to Cypress 12 we could only get a clean run of test results using Chrome 99. Now, using Cypress 12.3.0 and setting e2e.testIsolation to false, we are able to achieve a clean run with the latest version of Chrome (108).

[ClaasWilke]

So, rather astonishingly, our OOM issues have been completely tipped on their head simply by setting e2e.testIsolation to false. Prior to Cypress 12 we could only get a clean run of test results using Chrome 99. Now, using Cypress 12.3.0 and setting e2e.testIsolation to false, we are able to achieve a clean run with the latest version of Chrome (108).

Unfortunately, the e2e.testIsolation didn't help us. We are experiences OOM issues, when running our tests with both Cypress 11.2.0 and 12.3.0, both running with Electron (106).
Using Cypress 10.9.0 with Electron (102) worked just fine.

Besides, this issue seems to be related to (if not a duplicate of) #24905

[mschile]

The code for this is done in cypress-io/cypress#25462, but has yet to be released.
We'll update this issue and reference the changelog when it's released.

[kpturner]

The code for this is done in cypress-io/cypress#25462, but has yet to be released. We'll update this issue and reference the changelog when it's released.

What code are we talking about? Has the memory leak been found and fixed or are we relying on the setting that forces garbage collection? The latter didn't help (in our case).

[nagash77]

Hi @kpturner , during our investigation we found that Cypress was preventing chromium browsers from running their GC normally since there was no periods of inactivity. This new approach has Cypress monitor memory usage and calling for GC when appropriate. This does not address any potential memory leaks that may exist (we do not know of any current ones but that does not mean they can't exist). We are planning to focus on performance and resource utilization throughout 2023, so more to come on that front in the future.

If this new flag does not address your issue, I would ask that you please open a new ticket for your specific issue so that we can investigate that specific use case (which may have details that unearth some new findings). Thank you.

[kpturner]

Yes we tried the experimental garbage collection step - every 5 tests and indeed on every test and Chrome still crashed. For us, simply reverting to having e2e.testIsolation switched off resolved it.

I didn't see any other feedback on here to suggest that the garbage collection option helped others. Did I miss something? Closing the issue would suggest it is fixed for everyone else?

[hegeds]

from what I've been seeing this problem has multiple layers, one of those was gc not being able to do it's just. That is addressed in this PR.

An other thing that I found is that during long running tests that the command log is not being purged. This can be thought of a memory leak light, because it is an intended behaviour, just for a long run those entries can add up. This should be addressed in this ticket: #25230