fix: stats column binary_column has unsupported type binary

[omkar-foss]

Description

Handle BINARY column appropriately when specified in delta.dataSkippingStatsColumns while writing to a delta table.

Related Issue(s)

• closes Binary col in delta.dataSkippingStatsColumns not working #2854
• related Write struct with binary column failed with Json error: Binary is not supported by JSON #2373

Related PR(s)

• fix: ignore binary columns for stats generation #1766
• fix: avoid writing statistics for binary columns to fix JSON error #1498 (reverted in PR fix: revert premature merge of an attempted fix for binary column statistics #1544)

Documentation

N/A

[github-actions]

ACTION NEEDED

delta-rs follows the Conventional Commits specification for release automation.

The PR title and description are used as the merge commit message. Please update your PR title and description to match the specification.

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 72.08%. Comparing base (7d29885) to head (2ea21f4).

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #3146      +/-   ##
==========================================
- Coverage   72.08%   72.08%   -0.01%     
==========================================
  Files         134      134              
  Lines       43360    43360              
  Branches    43360    43360              
==========================================
- Hits        31258    31255       -3     
+ Misses      10085    10082       -3     
- Partials     2017     2023       +6     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[ion-elgreco]

@omkar-foss are you still working on a fix :)?

[omkar-foss]

@omkar-foss are you still working on a fix :)?

@ion-elgreco I was looking into it but haven't yet committed a fix as such. Looking into the UC issue right now, so you can pick this one up if you'd like :)

[omkar-foss]

I checked the delta lake protocol, and it looks like the stats are processed only for numeric columns types and stored in this structure: https://github.com/delta-io/delta/blob/master/PROTOCOL.md?plain=1#L1942-L1955

Since minValues and maxValues cannot be computed for BINARY columns, one option would be that we keep BINARY columns excluded from column statistics altogether.

Another option could be that we support BINARY columns and only compute numRecords and nullCount for them. minValues and maxValues will be unavailable.

@ion-elgreco any thoughts on which way we should go? My inclination is towards keeping them excluded.

[ion-elgreco]

@omkar-foss let's exclude it then yeah, maybe add a warm trace when such a column was set

[omkar-foss]

@omkar-foss let's exclude it then yeah, maybe add a warm trace when such a column was set

Yes sure, I'll update this PR accordingly. Thank you :)