Tags · IBM/spark-s3-shuffle

v0.9.6

CI: Add build for Spark 3.5.x

Signed-off-by: Pascal Spörri <psp@zurich.ibm.com>

Use move files using NIO if the shuffle dir is mounted as a file system.

Signed-off-by: Pascal Spörri <psp@zurich.ibm.com>

Revert: Modify the multipart upload size.

Signed-off-by: Pascal Spörri <psp@zurich.ibm.com>

Log Stage and Task ID to understand I/O bottlenecks.

Signed-off-by: Pascal Spörri <psp@zurich.ibm.com>

Use multiple threads for prefetching.

Signed-off-by: Pascal Spörri <psp@zurich.ibm.com>

Remove unused thread pool in S3ShuffleReader and cleanup unused imports.

Signed-off-by: Pascal Spörri <psp@zurich.ibm.com>

Enable tests if scala version 2.12.

Signed-off-by: Pascal Spörri <psp@zurich.ibm.com>

Buffer streams up to a threshold.

Signed-off-by: Pascal Spörri <psp@zurich.ibm.com>

Fix Scala 2.13 builds in SBT.

Signed-off-by: Pascal Spörri <psp@zurich.ibm.com>

Enable/disable caching of partition lengths with a configuration vari…

…able.

Signed-off-by: Pascal Spörri <psp@zurich.ibm.com>

PreviousNext