KV Read-after-Write Consistency

[ripienaar]

Proposed change

Since introducing Direct Get we lost our, admittedly imperfect, read-after-write consistency in KV stores.

We felt then that was an acceptable behaviour change but its clear that there are significant need for control in this area.

We might still default to the current behaviour but should at least provide a opt-in to be in-cluster consistent and in-supercluster eventually consistent.

This can take a few forms:

• We can add a leader-only direct get subject
• We can adopt some of the thoughts @MauriceVanVeen has around improving the consistency of the raft layer

Even if we do nothing, we should very clearly and prominently document what we do and do not support in places other than the ADR.

Use case

Increase the areas KV is applicable

Contribution

No response

[MauriceVanVeen]

We can add a leader-only direct get subject

If we'd be able to only get a response from a leader, we'd still not be able to guarantee read-after-write.
Without leader changes it'd work, but during every leader change there's a chance you get stale reads. Also if the current leader is network partitioned, and you write using a different connection to the new leader, you'd also get stale reads from the old (partitioned) leader.

There are a number of options to be able to guarantee this, even without requiring you to always connect directly to the leader. @mprimi has some thoughts about that as well, allowing replicas to answer if they can guarantee your read-after-write (with of course other tradeoffs as well).

Using the same connection object to guarantee read-after-writes sounds okay, but likely turns out to be brittle. Instead we should probably go for an implementation where we can strictly guarantee this. Regardless of if only the leader answers, or replicas can also answer.