Snapshot & Cursor for Volatile sources? #9207
-
|
Related to my previous question on rehydration from S3, I wonder if a nice feature would be to take snapshots of the current state, as well as the current offset in sources like Kafka and Kinesis to disk, such that during restarts Materialize only needs to backfill from the last snapshot. This would enable materialized views backed by volatile sources to survive restarts where the data has gone out of reach. |
Beta Was this translation helpful? Give feedback.
Replies: 1 comment 1 reply
-
|
Persistence is one of the biggest ongoing threads in Materialize, @danthegoodman1. For Kafka, we're already persisting offsets and timestamps in some capacity (this is what makes exactly-once sinks possible, for example). We'll start sharing more details into how the product is progressing with the wider community, so it should get easier for you to follow persistence work over the coming months! |
Beta Was this translation helpful? Give feedback.
-
|
Thanks, I am hoping very soon that restarts are a relative non-issue in the future once persistence is enabled for sources like Kafka (both offset and state). This is probably the single reason that prevents me from using Materialize right now. |
Beta Was this translation helpful? Give feedback.
Persistence is one of the biggest ongoing threads in Materialize, @danthegoodman1. For Kafka, we're already persisting offsets and timestamps in some capacity (this is what makes exactly-once sinks possible, for example).
We'll start sharing more details into how the product is progressing with the wider community, so it should get easier for you to follow persistence work over the coming months!