refactor: async writer + multi-part

[ion-elgreco]

Description

Changed to use arrows async writer and use multi-part uploads.

Related Issue(s)

• closes Use async writer + multipart + explore Datafusion sink #1984

[codecov]

Codecov Report

Attention: Patch coverage is 64.89362% with 33 lines in your changes missing coverage. Please review.

Project coverage is 72.17%. Comparing base (666179e) to head (d9e79c5).

Files with missing lines	Patch %	Lines
crates/core/src/operations/write/writer.rs	62.00%	8 Missing and 11 partials ⚠️
crates/core/src/operations/write/async_utils.rs	68.18%	13 Missing and 1 partial ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #3255      +/-   ##
==========================================
- Coverage   72.18%   72.17%   -0.01%     
==========================================
  Files         143      144       +1     
  Lines       45629    45709      +80     
  Branches    45629    45709      +80     
==========================================
+ Hits        32936    32992      +56     
- Misses      10621    10640      +19     
- Partials     2072     2077       +5     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[rtyler]

This is really such a major behavior change. I am not terribly familiar with the maturity level of multipart uploads in object_store I don't think this is necessarily a bad change, but I am doubtful of this addressing the originally linked issue.

As best as I can tell the buffers are still going to fill up memory until the flush, and then the flush is going to fan out to have parallel uploads

flowchart LR
    write --> buffer_batch;
    write --> buffer_batch;
    write --> buffer_batch;
    buffer_batch --> flush;
    flush --> p1;
    flush --> p2;
    flush --> p3;
    flush --> p4;
    p1 --> close;
    p2 --> close;
    p3 --> close;
    p4 --> close;

Loading

[ion-elgreco]

Good point, in its current form we indeed still buffer longer until the flush, but the buffering and writing has more parallelism now, so it should be faster.

We could do two things here btw:

• release this as is for people to experiment with (maybe after 1.0)
• iterate on this to also flush after the min - part size is available, also create some benchmark tests