S3 next level: offload, backup plus also data-sharing, querying?

[hpvd]

There are several discussions/issues about utilizing S3 for nats/jetstream

• Support S3 API for JetStream #5486
• S3 Compatibility for NATS Object Store #4871

=> What do you think of taking S3 -support directly to the next level,
by making early design choices to utilize it for
offload and backup
but also for data-sharing and querying?

Data-sharing and quering (3rd party usage) could be made possible by

1. storing messages in S3 directly in the standard open tablet format
   (-> parquet, see https://duckdb.org/2025/01/22/parquet-encodings.html)

2. one could read these parquet files from multiple S3 buckets e.g. via duckdb
   https://duckdb.org/docs/guides/network_cloud_storage/s3_import.html

3. easily combine them e.g. using DuckDB
   https://duckdb.org/docs/data/multiple_files/overview.html#parquet

4. combining different schemas by position or name if needed
   https://duckdb.org/docs/data/multiple_files/combining_schemas

5. and security could be addressed when using standard parquet encryption (Parquet Modular Encryption) specification
   https://duckdb.org/docs/data/parquet/encryption.html

raw idea inspired by
https://streamnative.io/blog/stream-table-duality-and-the-vision-of-enabling-data-sharing

[gedw99]

I like its simplicity and utility.

it’s this simple ?

Writes go to nats using nats, which then puts the stream changes into S3.

Reads are against s3 using http.

Many people have build these themselves.

The only thing will be reconciliation. If the nats streams change then what happens to S3. I guess nothing as that’s your migration ops layer where you decide .

[hpvd]

I like its simplicity and utility.

+1 on this!

it’s this simple ?

it doesn t change nats core. It can be an addition like KV-functionality.

Writes go to nats using nats, which then puts the stream changes into S3.

Reads are against s3 using http.

depends on your design. You can decide what to read from nats and what from DB e.g.only the reads benefitting from a DB e.g. complex queries

Many people have build these themselves.

good sign, there seem to be a need and we can bundle efforts :-)
having this as native addition, could benefit from deeper integration possibilities.

In the meantime for self-builders @jordan-rash pointed to https://github.com/synadia-io/connect

The only thing will be reconciliation. If the nats streams change then what happens to S3. I guess nothing as that’s your migration ops layer where you decide .

one could give different options/modes:

• backup(s) to specific point(s) of time,
• live reflection incl always all changes,
• offload on specific number of messages, size... see S3 Compatibility for NATS Object Store #4871 (comment)