Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Do not immediately abort the extraction if a seed chunk is invalid #220

Merged
merged 4 commits into from
Apr 18, 2022

Conversation

RyuzakiKK
Copy link
Contributor

  • Factor out writeChunk to a separate function

  • Do not immediately abort the extraction if a seed chunk is invalid

    If the seed points to a RW location it may happen that some files change
    while we are in the middle of an extraction.

    When using the "InvalidSeedActionRegenerate" option, we can try harder
    and attempt to take the invalid chunks from the self seed or the store.
    If both of those fail too, then we abort the entire operation.

  • Limit the maximum number of chunks is a single seed sequence

    If a seed is nearly completely equal to the output, we may end up with
    just a few SeedSequencer that are very long and others that are just
    one/a few chunks long.

    Because each SeedSequencer is handled by a goroutine (from a pool), we
    may reach a situation where the majority of the goroutines finish their
    operations and are just waiting for the longer SeedSequencer jobs to
    end.

    By limiting the maximum amount of chunks in the SeedSequencer, we will
    have jobs that are more balanced in term of amount of work.

  • When taking a chunk from the self seed, immediately return

    If we were able to take a chunk from the self seed, there is no need to
    continue looking into the existing file and the store.

@RyuzakiKK
Copy link
Contributor Author

I manually tested the situation where a seed changes while desync extract is in progress.

However I'd like to add some automated tests too. I'll probably look into it in the days.

Signed-off-by: Ludovico de Nittis <ludovico.denittis@collabora.com>
If the seed points to a RW location it may happen that some files change
while we are in the middle of an extraction.

When using the "InvalidSeedActionRegenerate" option, we can try harder
and attempt to take the invalid chunks from the self seed or the store.
If both of those fail too, then we abort the entire operation.

Signed-off-by: Ludovico de Nittis <ludovico.denittis@collabora.com>
If a seed is nearly completely equal to the output, we may end up with
just a few SeedSequencer that are very long and others that are just
one/a few chunks long.

Because each SeedSequencer is handled by a goroutine (from a pool), we
may reach a situation where the majority of the goroutines finish their
operations and are just waiting for the longer SeedSequencer jobs to
end.

By limiting the maximum amount of chunks in the SeedSequencer, we will
have jobs that are more balanced in term of amount of work.

Signed-off-by: Ludovico de Nittis <ludovico.denittis@collabora.com>
If we were able to take a chunk from the self seed, there is no need to
continue looking into the existing file and the store.

Signed-off-by: Ludovico de Nittis <ludovico.denittis@collabora.com>
@RyuzakiKK RyuzakiKK force-pushed the continue_with_invalid_seeds branch from 1f0831f to c3836d5 Compare April 18, 2022 08:08
@folbricht folbricht merged commit b545768 into folbricht:master Apr 18, 2022
@RyuzakiKK RyuzakiKK deleted the continue_with_invalid_seeds branch April 18, 2022 10:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants