fix(node): Merge from upstream - Delete consensus data from tables when it is no longer needed #5053
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
The latest updates on your projects. Learn more about Vercel for Git ↗︎ 4 Skipped Deployments
|
muXxer
approved these changes
Jan 28, 2025
piotrm50
approved these changes
Feb 3, 2025
gokhan-simsek-iota
force-pushed
the
node/delete-redundant-consensus-data
branch
from
891630f to
9f2a099
Compare
This PR is in preparation for data quarantining. The DQ PR will move the tables affected by this PR entirely into memory, and crash recovery will be driven by re-processing consensus commits. However, when we deploy DQ to a validator for the first time, we will restart in a state where uncertified consensus commits have already been marked as processed, so they will not be re-processed on startup. This means that upon starting up for the first time, some data will be on disk instead of in memory. There are two possible solutions to this: - All reads fall back to the database. This is a lot of ugly code, and is slow. - All data in the database is read into memory at startup. This is fast and simple. This PR bounds the amount of data we will have to read at startup in order to make the second option feasible.
gokhan-simsek-iota
force-pushed
the
node/delete-redundant-consensus-data
branch
from
9f2a099 to
7d26c6c
Compare
jkrvivian
pushed a commit
that referenced
this pull request
Feb 5, 2025
…ded (#5053) This PR is in preparation for data quarantining. The DQ PR will move the tables affected by this PR entirely into memory, and crash recovery will be driven by re-processing consensus commits. However, when we deploy DQ to a validator for the first time, we will restart in a state where uncertified consensus commits have already been marked as processed, so they will not be re-processed on startup. This means that upon starting up for the first time, some data will be on disk instead of in memory. There are two possible solutions to this: - All reads fall back to the database. This is a lot of ugly code, and is slow. - All data in the database is read into memory at startup. This is fast and simple. This PR bounds the amount of data we will have to read at startup in order to make the second option feasible.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description of change
Merging upstream changes: MystenLabs/sui@fedfb0d
From upstream:
This PR is in preparation for data quarantining. The DQ PR will move the tables affected by this PR entirely into memory, and crash recovery will be driven by re-processing consensus commits.
However, when we deploy DQ to a validator for the first time, we will restart in a state where uncertified consensus commits have already been marked as processed, so they will not be re-processed on startup.
This means that upon starting up for the first time, some data will be on disk instead of in memory. There are two possible solutions to this:
This PR bounds the amount of data we will have to read at startup in order to make the second option feasible.
Links to any relevant issues
Fixes: #4678
Type of change
Choose a type of change, and delete any options that are not relevant.
How the change has been tested
cargo run --release --bin iota start --force-regenesis --with-faucet
Change checklist
Tick the boxes that are relevant to your changes, and delete any items that are not.