Cypress sometimes stalls/hangs with no output when running in Jenkins with Docker #8206
Comments
|
Here's an abridged version of the full Cypress debug log: cypress-debug-abridged.log In our Jenkins output, this is the last message I see: You can see this mocha test at the top of the logfile. But after that, no more mocha tests are triggered for some reason. Nothing runs out of memory, nothing crashes, nothing fails, nothing exits. Everything just stops responding and Cypress seems to be waiting for an event that'll never happen. I can furnish the full debug logfile (13.2 MB) upon request. |
|
We are facing similar issue, Cypress just hangs indefinitely during UI test runs, no error, no logs. The only option is to close and run the tests again. We are running them locally, have not integrated with CI yet. |
|
Also seeing this, about 1 every 50 test runs, Cypress will hang, occasionally for 12 hours(!) if no one notices |
|
The debug logs don't indicate anything that we can go off of. We'll need a way to reproduce this in order to investigate. |
jennifer-shehane
added
the
stage: needs information
|
I'm also experiencing the same issue. It seems that it's a recent issue. At least I was able to experience it with Cypress 4.11 and also 4.12.X. I even had to write down a script that cancels a test if it's running for more than 30 min and trigger it again. In our particular case we have been running 3 instances of Cypress in parallel in the same machine and it's been working for months. But now, as someone said, from time to time one of the tests hangs indefinitely, with no output indicating if there was a failure or something. I will try to gather some info of what's going on and post it here. |
Run multi-hour test suites in a Jenkins/Docker environment until you start to see Cypress hanging. 😄 On a serious note, it sounds like several people are seeing this problem, and we all have access to such environments, so maybe you'll have to just leverage that rather than trying to repro yourself. Is there any other logging or debugging instrumentation we should turn on that would be helpful? |
|
I've also experienced this. After a few days of trying things out, making these 3 changes helped me.
Please note I'm just sharing what seemed to have worked for me, but I don't consider this a solution. It sounds like an issue that should be looked into. |
|
Interesting. To the best of my knowledge, we've seen the opposite trend: using Electron was worse and switching to Chrome has slightly improved things. But we still run into this hanging problem with both browsers. (This would make sense if the problem is actually with mocha, which is kind of what the logs would suggest.) |
|
FWIW, we were seeing this nearly every test suite execution (>90%) in our CI runner (AWS CodeBuild with parallelization, using Chrome), with no issues running locally. Today we upgraded to v5.0.0 and the hanging/stalling has seemingly stopped |
|
@bradyolson Good to know! We're attempting to upgrade Cypress this week, so maybe in a few days I can report back with our results. |
|
I have an update on this: I noticed that Cypress tests were downloading a few static assets in a continuous stream. The site logo, a font, etc. They would be downloaded hundreds of times in a single test, and in some cases completely overwhelm the browser, causing the test to fail. This happened locally too. I downgraded to version 4.9.0 from 4.12.0 and the issue went away. I hope this gives some context to help. For some other context: this happened in end-to-end mode and in mock mode (we run our tests in both modes) AND in a new mode using mock service worker, but NOT when running the app in development or in production through a regular Chrome browser. |
|
We finally got Cypress 5 to run and hit the same exact problem. No improvement. |
|
I spent the afternoon trying to run our tests with Firefox (which I realize is in "beta") and noticed that this problem seems a lot worse with Firefox. It's the same symptoms as before: console output stops appearing, mocha events appear to stop coming in, but the debug log keeps chugging along with memory and CPU updates. When trying to run our usual battery of ~50 test suites, we couldn't get further than 5 suites, sometimes failing at even the first one. With Chrome, we could always get at least 25 or 30 suites in before the stall happens. |
|
I've tried a new approach/workaround in our project where instead of passing all our test suites to Cypress all at once, I pass the test suites one at a time. Before: After: This seems to have helped a little bit, though last night we hit an error even with this approach. About 2 hours into the testing, a test suite stalled after running for only a couple minutes and Jenkins had to kill the task a couple hours later. |
|
We encountered this failure again last night with the same symptoms. Now that we're running our test suites one at a time, the logs are smaller and easier to digest. Still seems like Cypress is at fault here. In the Jenkins output, we can see that the test suite stalled after executing for only a minute: In the full debug logfile, I can see that this is the last event we get from mocha: There are no more mocha events and the test never starts execution. This is also verified by the video file that Cypress writes out. I can see in the video that the sidebar never "expands" for the doomed test. Instead of seeing the list of I can furnish the Jenkins log, Cypress debug log, and Cypress video upon request (in a DM or some other private channel). |
|
We have the same issue and we are using Cypress 5.2.0 as it seems to bring in several performance fixes. We are now using Cypress since the 2.X days and I feel that this is getting worse over time. |
|
Any update on this issue? |
+1. Also waiting for updates. We continue to run into this issue often, even with the latest version (5.2.0). I've offered to furnish full logs, but no one's taken me up on it. Kind of feels like the Cypress folks are hoping this issue will just magically disappear. |
|
We are facing the same issue. Tried to update to 5.2.0 and all kind of workarounds but this is still happening. |
|
Similar issues here. There has been a runner change as well as upgrade to Cypress 5.2.0 . We have parallelization as well, and we are using the cypress included docker image. Firefox tests hang indefinitely. With Electron, we can get the same crashes, but rarely the tests can work. The trends seems to be tests with ui-login. For the time being we disabled them in CI. This might help someone. Then, in the test wrap any describe/context/it block |
|
If you are running Chrome or Chromium it can be more useful to use https://developers.google.com/web/tools/puppeteer/troubleshooting#tips |
This helped. Thank you! The info resource can also be found that at Cypress docs. If your tests are hanging in the CI, take a look. There are plenty of questions about configuring the const fs = require('fs-extra')
const path = require('path')
// tasks go here
const percyHealthCheck = require('@percy/cypress/task')
const mailosaurTasks = require('./mailosaur-tasks')
// merge all tasks
const all = Object.assign({},
percyHealthCheck,
mailosaurTasks
)
function getConfigurationByFile(file) {
const pathToConfigFile = path.resolve('cypress/config', `${file}.json`)
return fs.readJson(pathToConfigFile)
}
module.exports = (on, config) => {
on("task", all);
// needed to address issues related to tests hanging in the CI
on('before:browser:launch', (browser = {}, launchOptions) => {
if (browser.family === 'chromium' && browser.name !== 'electron') {
launchOptions.args.push('--disable-dev-shm-usage')
}
return launchOptions
});
const file = config.env.configFile || 'dev'
return getConfigurationByFile(file)
} |
|
Similar issue here. Already throughout months. From time to time (several times in a week) the execution just stops on random tests. No errors, no memory ran out. Just stopped and JENKINS continues its execution for "dead tests". We checked a Server, Docker, Jenkins, and nothing outstanding there. Can't reproduce it stably nohow, it happens periodically. |
|
@bahmutov @jennifer-shehane Can anyone from Cypress team update on this? |
|
Guys me to facing same issue. I have been using cypress for almost 1 yr 4 months never faced this issue, I updated to latest version then it keeps hanging or some time memory issues in local and in Circle CI too. Badly need a fix... -G |
|
I have spotted this also in the CircleCI jobs. |
|
It is affecting us badly lately. It used to be a rare case, but seems to have become worse for the last two months or so. The debug output (via Looks like this: The last messages above are from that periodic network request (from a PerimeterX sensor). Nothing test-specific is printed after that, just Perimeterx requests interleaved with CPU profiling Cypress 12.11 with Electron |
|
🔔 Reposting this for visibility as it might get have gotten buried under the other comments. We use this script to run our Cypress tests: https://gist.github.com/martincarstens/70dd91176e420c6c9ca3de96cafac71d It's a simple script that auto detects hangs and prevents runaway jobs. Since implementing it over a year ago, Cypress has been stable for us. We have 157 spec files running in various configurations in GitHub Actions with parallel jobs. We are using the latest version of Cypress. To be fair, we shouldn't need to be using this script, but it is the only thing that returned sanity to our workflows. @stuart-clark might be able to provide some updates running Cypress with the script in Circle CI. @cosmith any updates from your side? Rooting for the Cypress team to put this one to bed. |
|
Hello, on our side we migrated to Playwright. I do remember that in the end the "no output" that we were seeing in CircleCI was actually a "very long output" when some tests were failing and retrying several times in a row. So it was not really a bug in Cypress (even though it would have been a lot easier to debug if there was some log stating that the test was restarting). |
We've also started migrating to Playwright, but it's taking us a bit because of all the effort invested into writing Cypress specs. Thanks for the update. |
|
This bug is very breaking for my use case, so I'm studying a soft migration to Playwright now as well, at least the non-working tests for now. We also have a big problem caused by how Cypress handles cross-origin and iframe tests, which unfortunately is not stable enough right now, but some test cases are very dependant on it. This is a real bummer, since the last couple of months I spent on actually migrating our tests from another platform to Cypress, so finding out some very crucial steps are impossible is really demotivating. For the others watching the thread, some links I read on this migration: Migration from Cypress to Playwright - hype or great? On Migrating from Cypress to Playwright Migrating from Cypress to Playwright Rewriting tests from Cypress to Playwright using GPT3 by Gajus Kuizinas Convert Cypress to Playwright (tool built for this migration) Hopefully the Cypress team fixes these crucial things eventually, but reading over a year old (in this case, almost three) posts reporting the same problem to something as important as just running tests, in a test framework, and that functionality not working, is a deal-breaker. |
|
also affecting me in GitLab CI. Using the |
|
Any updates on this? |
|
We ended up patching the runner (we currently use |
|
TLDR: Add I have an observation that could assist the Cypress team in fixing a potential bug in Cypress (if confirmed as a bug), or individuals like myself in applying a temporary workaround to address the issue. I came across this thread when my test suite began encountering issues during execution in the CI environment, sometimes stalling midway and necessitating a restart. While restarting occasionally resolved the problem, it wasn't a consistent solution. Despite my attempts at investigation, I was unable to pinpoint the root cause. Subsequently, I upgraded Cypress to the latest version and encountered a similar issue with a very straightforward test involving the The I speculated that the issue might be related to requests spawned by the main view of the application, to which users are redirected after a successful sign-in. As a quick workaround, I appended However, I tried to find a more elegant solution utilizing Consequently, I am currently relying on the slightly inelegant but functional |
|
Thanks for this valuable input!!!
…On Sun, 20 Aug 2023 at 17:08, Ihor Dotsenko ***@***.***> wrote:
*TLDR:* Add cy.wait(1000) or more at the end of the test to wait for all
requests to finish.
I have an observation that could assist the Cypress team in fixing a
potential bug in Cypress (if confirmed as a bug), or individuals like
myself in applying a temporary workaround to address the issue.
I came across this thread when my test suite began encountering issues
during execution in the CI environment, sometimes stalling midway and
necessitating a restart. While restarting occasionally resolved the
problem, it wasn't a consistent solution. Despite my attempts at
investigation, I was unable to pinpoint the root cause.
Subsequently, I upgraded Cypress to the latest version and encountered a
similar issue with a very straightforward test involving the Sign In
form. This test includes visiting localhost, inputting login credentials,
submitting the form, and then verifying the redirected URL.
The sign-in.cy.ts file wouldn't complete execution in Cypress. It failed
to display the statistics for the passed/failed/skipped tests; the
process simply hung. Given the simplicity of the test, it became evident
that the problem lay with Cypress itself.
I speculated that the issue might be related to requests spawned by the
main view of the application, to which users are redirected after a
successful sign-in. As a quick workaround, I appended cy.wait(1000) to
the test's end, and to my delight, it resolved the problem 🎉
However, I tried to find a more elegant solution utilizing
cy.intercept('/api/**').as('api') followed by ***@***.***').
Regrettably, this approach didn't yield the desired outcome.
Consequently, I am currently relying on the slightly inelegant but
functional cy.wait(1000) workaround, with the hope that this issue will
eventually be addressed within Cypress itself. 💡 It's possible that all
pending requests could be automatically canceled at the end of each test...
—
Reply to this email directly, view it on GitHub
<#8206 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AFKDNFFVS6TKELSK42WQJ5LXWH42HANCNFSM4PWYC6YA>
.
You are receiving this because you commented.Message ID:
***@***.***>
|
Hi Martin! 👋 To clarify, we've always used Github Actions, not Circle CI. We don't run into this issue anymore. So we are stable, some notes on our configuration off the top of my head:
While I still think there are issues here, we get around them by being matrixed in such a way that a runner is only running 3-6 spec files of very few To do this efficiently (💸) on Github Actions one needs to leverage For anyone facing this, I would really just suggest breaking your tests up as much as possible and using a matrix strategy or a script similar to what @martincarstens wrote. |
|
Thanks @GarrisonD We discovered this today as part of using the new Test Replay feature with Cypress v13. There was always a 1 min wait in our after hook after an assertion around network responses failed. It is definitely far from ideal adding a wait in like this. There is clearly a bug here around disposing of network call promises or something perhaps |
|
@GarrisonD <https://github.com/GarrisonD>
You mean add afterEach?
…On Sun, 20 Aug 2023 at 16:08, Ihor Dotsenko ***@***.***> wrote:
*TLDR:* Add cy.wait(1000) or more at the end of the test to wait for all
requests to finish.
I have an observation that could assist the Cypress team in fixing a
potential bug in Cypress (if confirmed as a bug), or individuals like
myself in applying a temporary workaround to address the issue.
I came across this thread when my test suite began encountering issues
during execution in the CI environment, sometimes stalling midway and
necessitating a restart. While restarting occasionally resolved the
problem, it wasn't a consistent solution. Despite my attempts at
investigation, I was unable to pinpoint the root cause.
Subsequently, I upgraded Cypress to the latest version and encountered a
similar issue with a very straightforward test involving the Sign In
form. This test includes visiting localhost, inputting login credentials,
submitting the form, and then verifying the redirected URL.
The sign-in.cy.ts file wouldn't complete execution in Cypress. It failed
to display the statistics for the passed/failed/skipped tests; the
process simply hung. Given the simplicity of the test, it became evident
that the problem lay with Cypress itself.
I speculated that the issue might be related to requests spawned by the
main view of the application, to which users are redirected after a
successful sign-in. As a quick workaround, I appended cy.wait(1000) to
the test's end, and to my delight, it resolved the problem 🎉
However, I tried to find a more elegant solution utilizing
cy.intercept('/api/**').as('api') followed by ***@***.***').
Regrettably, this approach didn't yield the desired outcome.
Consequently, I am currently relying on the slightly inelegant but
functional cy.wait(1000) workaround, with the hope that this issue will
eventually be addressed within Cypress itself. 💡 It's possible that all
pending requests could be automatically canceled at the end of each test...
—
Reply to this email directly, view it on GitHub
<#8206 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AFKDNFFVS6TKELSK42WQJ5LXWH42HANCNFSM4PWYC6YA>
.
You are receiving this because you commented.Message ID:
***@***.***>
|
|
@bahunov I have more than one test in a file, so yeah, I use The key point here is to wait for running requests, where and how is a minor thing. |
|
Just want to share one more workaround for hang problem. In our case we are running files individually, and hang occured after test completion - for some reason cypress failed to properly exit. I ended up with implementing parallel watchdog with bash to track this. Gist: https://gist.github.com/zeldigas/6d2041c51b96b98b4859017f55a9bc34 |
|
@Phonesis Please make sure to update to the latest version of the Cypress App. We've been actively addressing hanging issues related to Test Replay. |
|
Hi everyone, stalling and hanging within Cypress could be attributed to many different reasons. We do work through hangs/stalls issues as quickly as we can since we know this is the worst situation that your tests can be stuck in and it is an extra expense for CI. The causes we've found for hanging have been very wide and we do need clear reproducible examples provided in order to investigate. I suggest opening a new issue describing any current hanging issues and as much info as you can provide for us to investigate, including debug logs. Everyone's issue in this thread could be caused by very different things and unrelated. I'll be closing this issue since it is not encompassing a single cause that we can work towards a resolution on and close. We have fixed several causes of hanging issues over the years that might have addressed many of these as well. |
How would one know the root cause for their specific case? |
|
I have also this issue. |
|
@madardour please open a new issue with a reproducible example and the Cypress team will be happy to investigate. |
jennifer-shehane
added
CI
|
I have been running into a similar issue with Cypress just hanging. In my case it was on a very particular new test suite that was added. What we found is that the app UI had performance issues with excessive DOM element counts. Hope that helps with reproduction and fixing this scenario. Cypress We have ~4500 DOM elements on initial render and that number kept growing (anything over ~1500 is known to be a possible perf bottleneck) |
jennifer-shehane
removed
the
prevent-stale
and others
Current behavior:
I feel like this kind of issue has been referenced many times before, but even after trying all the solutions I've found, we still continue to run into this problem. We've been seeing this issue for many months now, with early versions of Cypress 4.x and even with the most recent version today.
Long-running (multi-hour) test suites often "stall": the output stops appearing, but nothing actually fails or crashes. We actually have to rely on the timeout mechanism in Jenkins to kill the task because Cypress is just stuck.
I have enabled debug logs and honestly I don't see anything helpful in them. I'll upload the relevant portion next, but there are no mentions of failures, crashes, or even being out of memory. We're already following all the guidance in the Cypress docs about disabling /dev/shm and using the correct IPC setting.
Desired behavior:
Tests should either run to completion or Cypress should fail with a clear error.
Versions
Cypress: 4.12.1
OS: Linux
Browser: Chrome 83 (headless)
The text was updated successfully, but these errors were encountered: