Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix(ci): Revert enabling file-based persistence on dev server for stress tests #1497

Merged
merged 2 commits into from
Aug 30, 2024

Conversation

mjameswh
Copy link
Contributor

What changed

  • Disable file-based persistence on dev server for stress tests, which we previously enabled in fix(ci): Enable file-based persistence on dev server for stress tests #1480. Though desirable in some aspects, enabling file-based persistence resulted in reducing tests performance, causing recurring timeout errors during nightly stress tests, as well as flakes during regular PR/main branch commit stress tests.

  • For now, we can hopefully resolve the original issue by scaling up the GHA runners (which we already did in chore: bump GHA runner for stress tests #1492). As a more stable solution, we can investigate at transitioning to Temporal Cloud.

@mjameswh mjameswh requested a review from a team as a code owner August 14, 2024 22:03
Copy link
Member

@Sushisource Sushisource left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Wow, fascinating.

Is the issue here basically a bunch of writes are hitting disk instead of memory and that just slowed down the server a bunch?

Good on you for figuring that one out. I needed to be a little more permissive in terms of looking outside of the commit range where it started failing, which I did a little, but not enough, lol.

@mjameswh mjameswh merged commit 067b5d7 into main Aug 30, 2024
86 checks passed
@mjameswh mjameswh deleted the revert-1480-fix-nightly branch August 30, 2024 17:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants