Improve compaction testing

[hzxa21]

Recently we have witnessed multiple issues related to compaction/compactor:

• bug: compaction task stuck for hours #10209
• bug: compaction schedule loop can be stuck when there is compactor failure #10161
• meta node panicked at 'should find the level to insert into when applying compaction generated delta' #10538
• bug(compaction group): expired task is not correctly rejected #10215
• bug(meta): meta-node crash because of apply version delta failed #10240

These issues are related to excessive compaction load, compactor failure and compaction group split. Some of these issues are corner cases that can hardly be caught while others are not. Although we have compaction running in different testing pipelines (longevity, perf, main-cron, ut, e2e) to cover the basic cases, we have low coverage on these scenarios.

Some ideas:

• [Enable injecting compactor/meta failure and restarts in chaos-mesh test
• [Put more loads on L0 in the testing pipeline (originally proposed by @Li0k)
  • We can leverge deterministic compaction test to generate more L0 SSTs to reply before triggering compaction.
  • We can shutdown all compactors in the longevity/chaos-mesh pipeline and let L0 SSTs to accumulate for a while before restarting the compactors.
• Can we also test compaction group split via the deterministic compaction test tool?

Open to other ideas as well.

[hzxa21]

cc @Li0k @Little-Wallace

[fuyufjh]

Hey, any updates on this?

[Li0k]

Hey, any updates on this?

No progress so far

[github-actions]

This issue has been open for 60 days with no activity.

If you think it is still relevant today, and needs to be done in the near future, you can comment to update the status, or just manually remove the no-issue-activity label.

You can also confidently close this issue as not planned to keep our backlog clean.
Don't worry if you think the issue is still valuable to continue in the future.
It's searchable and can be reopened when it's time. 😄