integrated storage - leader issues

[daipok]

I was trying to understand, how does an election occur.

I tested in a cluster of 3 nodes, and tried both taking the leader down or a bao operator step-down, both work reasonably well.

What if however, there are some issues with the leader node. I did try to use iptable to drop packets on 8200 and 8201 (listening port and cluster port), and try to simulate some sort of failure pattern, but I found that this does not trigger a leadership shift.

Furthermore, I cannot even do a list-peers from the follower. Does the bao call actually gets forwarded to the leader?

[cipherboy]

@daipok Yes, that forwarding behavior is documented here: https://openbao.org/docs/internals/high-availability/

Only the active server processes all requests; the standby server redirects all requests to an active OpenBao server.

So until the active node's heartbeat loss (from iptables filtering) drops below the threshhold (controlled by the performance_multiplier parameter of raft), you won't be able to connect to the cluster.

Depending on your environment, tuning performance_multiplier below 5 might be a viable option, else we're working towards horizontal scalability soon and thus will have standbys able to service read requests.

[daipok]

I am not sure if I read this right. If I dont set the performance_multiplier it should be defaulted to 0 that is the lower performance. So supposedly it will just take longer to detect a heartbeat loss? So setting to 1 (or something less than 4) increase the frequency of the check?

For the heartbeat check is that checking port 8200 or 8201?

[cipherboy]

performance_multipler defaults to 0, to mean "whatever the developers think is best at the moment", which correlates to a library value of 5 (for now). Lower is "faster systems", higher is "slower systems".

So yes, setting to (1...4) would make the check occur more frequently, at the risk of potentially changing leadership too frequently, unnecessarily.

Because this heartbeat is part of raft, it'll occur on port 8201.

[daipok]

I am testing performance_multiplier = 1 at the moment.

I found that on the follower node it has
[DEBUG] core: forwarding: error sending echo request to active node: error="rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp 10.198.124.11:8201: i/o timeout""

so I suppose it's losing the "heartbeat" to the leader node at 10.198.124.11

But this does not shift the leadership yet for some reason. The leader is still at 10.198.124.11. So I suspect I am missing some configuration to make this magic work

[daipok]

I am not sure if I follow correctly.

I set performance multiplier to 1, and I don't see the heartbeat becoming faster, it's still about 5s apart.
storage "raft" { ... performance_multiplier =1 }

Mar 18 19:18:11 node2.obao [627460]: 2025-03-18T19:18:11.184Z [DEBUG] core: forwarding: error sending echo request to active node: error="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial tcp 10.198.124.11:8201: i/o timeout\"" Mar 18 19:18:16 node2.obao bao[627460]: 2025-03-18T19:18:16.184Z [DEBUG] core: forwarding: error sending echo request to active node: error="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial tcp 10.198.124.11:8201: i/o timeout\"" Mar 18 19:18:21 node2.obao bao[627460]: 2025-03-18T19:18:21.184Z [DEBUG] core: forwarding: error sending echo request to active node: error="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial tcp 10.198.124.11:8201: i/o timeout\""

Another interesting thing is, I don't see an re-election happening. If the follower cannot reach the leader through heartbeat, does it really guarantee a re-election?

[cipherboy]

@daipok Hmm, do you see a line like:

2025-03-18T15:06:09.697-0500 [TRACE] storage.raft: applying raft config: inputs="map[path:/home/ascheel/.local/share/devbao/nodes/prod/storage/raft performance_multiplier:1]"
2025-03-18T15:06:09.699-0500 [TRACE] storage.raft: reloaded raft config to set lower timeouts: config="raft.ReloadableConfig{TrailingLogs:0x2800, SnapshotInterval:120000000000, SnapshotThreshold:0x2000, HeartbeatTimeout:1000000000, ElectionTimeout:1000000000}"

And then later:

2025-03-18T15:06:09.699-0500 [TRACE] storage.raft: reloaded raft config to set lower timeouts: config="raft.ReloadableConfig{TrailingLogs:0x2800, SnapshotInterval:120000000000, SnapshotThreshold:0x2000, HeartbeatTimeout:1000000000, ElectionTimeout:1000000000}"

Note that you'll have to have unsealed the node for it to join the cluster.