Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Tracking issue for supporting shard level replay #412

Closed
1 of 10 tasks
Rachelint opened this issue Nov 18, 2022 · 1 comment
Closed
1 of 10 tasks

Tracking issue for supporting shard level replay #412

Rachelint opened this issue Nov 18, 2022 · 1 comment
Labels
A-analytic-engine Area: Analytic Engine A-WAL Area: Write ahead log layer feature New feature or request tracking issue Issue tracks progress for something

Comments

@Rachelint
Copy link
Contributor

Rachelint commented Nov 18, 2022 •
edited
Loading

Describe This Problem

Table level replay is too slow, so we try to support shard level replay to relax this problem.
It is a big work, so I split it into some phases and track its progress here.

Proposal

  • complete unimplemented scan method in all wal implementations.
    feat: refacor existing wal implementations to support scanning logs of shard #400
  • implement wal adapter to keep the reading view of table level while using(and the actual reading work is doing in shard level).
    • implement the read buffer.
    • implement the wal adapter.
  • refactor the original recovering process to make use of the wal adapter.
    • refactor the original recovering process in standalone mode.
    • add Replayer to carry out the recovering work.
  • bug fixs:
    • recover by flushed_sequence in manifest.
    • modify RegionId to a struct including ShardId and its version.

Additional Context

No response
@Rachelint Rachelint added feature New feature or request A-analytic-engine Area: Analytic Engine tracking issue Issue tracks progress for something labels Nov 18, 2022
@jiacai2050 jiacai2050 added the A-WAL Area: Write ahead log layer label Mar 6, 2023
@jiacai2050
Copy link
Contributor

Fixed in #799

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
A-analytic-engine Area: Analytic Engine A-WAL Area: Write ahead log layer feature New feature or request tracking issue Issue tracks progress for something
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@jiacai2050 @Rachelint