Log indicators in their own dedicated chunks #8768
Labels
🔩 data model
Sorbet
🪵 Log & send APIs
Affects the user-facing API for all languages
🚀 performance
Optimization, memory use, etc
⛃ re_datastore
affects the datastore itself
Indicators are meant to go, but this won't happen overnight. In fact, they will be here for the foreseeable future.
In the meantime, it would be nice to reduce their negative impact, if possible (i.e. with very minimal efforts).
There are two problems with them that I want to focus on in this issue.
First problem: on the standard row-oriented log path, indicators actually consume 4 bytes of space per row in each chunk.
The reason for that is that chunks are, at the moment, only capable of carrying
ListArray
s around (we want to be able to use native-typed arrays as columns in the future, but this is not a small change and this won't happen for a long time).Corollary: each time you log an indicator (i.e. each time you log an archetype), the indicator's
NullArray
gets wrapped in aListArray
, and that list must have N offset values where N in the number of rows in the chunk. The null array itself doesn't take space (well it takes a fixed 4 bytes), but the outer list's offsets do.Second problem: all of that is just as true on the column-oriented path, and becomes even more relevant today with new
send_columns
APIs that actually enforce that the appropriate indicators are logged.We should investigate the possibility of always logging indicators into their own little chunk, on both the row- and column- oriented paths.
The text was updated successfully, but these errors were encountered: