Conditional overwrite into Delta Lake Table

[datanikkthegreek]

Is your feature request related to a problem?

No

Describe the solution you'd like

Currently, the delta lake implementation in daft allows only a complete overwrite but not a conditional overwrite based on a predicate.

Delta on Spark supports this already for some time: https://delta.io/blog/delta-lake-replacewhere/

I also see that the rust delta lake implementation supports the "predicate" options https://delta-io.github.io/delta-rs/api/delta_writer/#deltalake.write_deltalake.

Have you considered this besides implementing delete, update and merge?

Describe alternatives you've considered

The current alternative would be transforming into a pyarrow and then use the python deltalake library

Additional Context

No response

Would you like to implement a fix?

No

[universalmind303]

@datanikkthegreek that seems like it'd be a great addition that should be relatively easy to add.

The only concern I have is that the predicate takes in a SQL string?? that may not be compatible with our SQL dialect. (IIRC, delta-io uses datafusion internally).