Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Stability of parallel compression #2238

Closed
marxin opened this issue Jul 9, 2020 · 7 comments
Closed

Stability of parallel compression #2238

marxin opened this issue Jul 9, 2020 · 7 comments
Labels
question

Comments

@marxin
Copy link
Contributor

marxin commented Jul 9, 2020

Just to be sure. When using the parallel compression, is the compressed stream stable?
I mean can it vary when compression is repeated or is it deterministic?
@marxin marxin mentioned this issue Jul 9, 2020
Merged
@Cyan4973
Copy link
Contributor

Cyan4973 commented Jul 9, 2020

Compression is reproducible.

@terrelln
Copy link
Contributor

terrelln commented Jul 9, 2020

The single-threaded output will be different than the multi-threaded output. However, both are deterministic, and the multi-threaded output produces the same compressed data no matter how many threads you use.

@marxin
Copy link
Contributor Author

marxin commented Jul 10, 2020

Great, good job made!

@marxin marxin closed this as completed Jul 10, 2020
@marxin
Copy link
Contributor Author

marxin commented Jul 23, 2020

One further question: is the compressed output dependant on a number of threads that are used for compression?

@terrelln
Copy link
Contributor

One further question: is the compressed output dependant on a number of threads that are used for compression?

No. Single-threaded output is different than multi-threaded output, but the multi-threaded output is the same for any number of threads.

@ghost
Copy link

ghost commented Sep 25, 2020 •
edited by ghost
Loading

the multi-threaded output is the same for any number of threads.

When:

  • use ZSTD_e_end end directive
  • output buffer size >= ZSTD_compressBound()

The job number is calculated by ZSTDMT_computeNbJobs() function.

Then in this case, different thread number may produce different compressed data, is this intended?

@terrelln terrelln mentioned this issue Sep 25, 2020
Closed
2 tasks
@terrelln
Copy link
Contributor

Then in this case, different thread number may produce different compressed data, is this intended?

No. I'd call that a bug. I've opened Issue #2327 to track it.

If you need to work around this bug, don't start your streaming job with ZSTD_e_end. Pass at least one byte of input with ZSTD_e_continue before calling ZSTD_e_end.

@codicodi codicodi mentioned this issue Oct 7, 2020
Merged
MaskRay added a commit to llvm/llvm-project that referenced this issue Sep 21, 2022
@MaskRay
[ELF] Parallelize --compress-debug-sections=zstd
fa74144 
See D117853: compressing debug sections is a bottleneck and therefore it
has a large value parallizing the step.

zstd provides multi-threading API and the output is deterministic even with
different numbers of threads (see facebook/zstd#2238).
Therefore we can leverage it instead of using the pigz-style sharding approach.

Also, switch to the default compression level 3. The current level 5
is significantly slower without providing justifying size benefit.

```
  'dash b.sh 1' ran
    1.05 ± 0.01 times faster than 'dash b.sh 3'
    1.18 ± 0.01 times faster than 'dash b.sh 4'
    1.29 ± 0.02 times faster than 'dash b.sh 5'

level=1 size: 358946945
level=3 size: 309002145
level=4 size: 307693204
level=5 size: 297828315
```

Reviewed By: andrewng, peter.smith

Differential Revision: https://reviews.llvm.org/D133679
veselypeta pushed a commit to veselypeta/cherillvm that referenced this issue May 20, 2024
@MaskRay @veselypeta
[ELF] Parallelize --compress-debug-sections=zstd
ae04069 
See D117853: compressing debug sections is a bottleneck and therefore it
has a large value parallizing the step.

zstd provides multi-threading API and the output is deterministic even with
different numbers of threads (see facebook/zstd#2238).
Therefore we can leverage it instead of using the pigz-style sharding approach.

Also, switch to the default compression level 3. The current level 5
is significantly slower without providing justifying size benefit.

```
  'dash b.sh 1' ran
    1.05 ± 0.01 times faster than 'dash b.sh 3'
    1.18 ± 0.01 times faster than 'dash b.sh 4'
    1.29 ± 0.02 times faster than 'dash b.sh 5'

level=1 size: 358946945
level=3 size: 309002145
level=4 size: 307693204
level=5 size: 297828315
```

Reviewed By: andrewng, peter.smith

Differential Revision: https://reviews.llvm.org/D133679
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
question
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@Cyan4973 @marxin @terrelln