fix(ci): Revert enabling file-based persistence on dev server for stress tests #1497
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…ss tests…" This reverts commit 6b050a6.
Sushisource
approved these changes
Aug 15, 2024
There was a problem hiding this comment.
Wow, fascinating.
Is the issue here basically a bunch of writes are hitting disk instead of memory and that just slowed down the server a bunch?
Good on you for figuring that one out. I needed to be a little more permissive in terms of looking outside of the commit range where it started failing, which I did a little, but not enough, lol.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What changed
Disable file-based persistence on dev server for stress tests, which we previously enabled in fix(ci): Enable file-based persistence on dev server for stress tests #1480. Though desirable in some aspects, enabling file-based persistence resulted in reducing tests performance, causing recurring timeout errors during nightly stress tests, as well as flakes during regular PR/main branch commit stress tests.
For now, we can hopefully resolve the original issue by scaling up the GHA runners (which we already did in chore: bump GHA runner for stress tests #1492). As a more stable solution, we can investigate at transitioning to Temporal Cloud.