Could not discover and join another node. Proceeding as a standalone node - wait a minute![Suggestion]

[SimonUnge]

RabbitMQ series

4.0.x

Operating system (distribution) used

*nix

How is RabbitMQ deployed?

Community Docker image

What would you like to suggest for a future version of RabbitMQ?

Hey,

Lets say we have a situation where we have a cluster of 3 nodes, A, B and C, that is up and running, and clustered with peer discovery.
For whatever unfortunate reason, Node A is restarted, with its state whipped, i.e a 'fresh' start for Node A. Node A is trying to join the configured cluster, in its eyes, for the first time. Node B and C disagrees, as they think Node A is already a member.
Node A will retry, by default, 30 times. If it ends up failing, it will proceed as a standalone node. Lets throw some fuel on the fire, and have a load balancer infront of these 3 nodes, who only sees these 3 nodes as up and running, and assumes they are clustered. To end up here there ofc needs to be a series of unfortunate events, and failures in external monitoring systems to catch this issue. But still.

I wonder if there has been any thoughts about having configuration to not allowing a node to start if it fails to join a configured cluster using peer discovery, and after the 30 (default) retries, stop the node from starting at all, to avoid a situation where we have '2' clusters started, instead of 1?

[michaelklishin]

@SimonUnge it was discussed. There wasn't much opposition to it but then the focus has shifted to #13050 and this particular change went nowhere.

[SimonUnge]

@michaelklishin Oh ok, interesting. So, basically the default would now be unlimited retires? This might actually be 'good enough', as one should have mechanisms that notices that a node fails to come up and the logs would give us the heads up on why.

If my understanding is correct, I think we can close this one as answered :)

[michaelklishin]

Probably. @mkuratczyk is on PTO right now, so we will know in a week or two (I don't know when exactly).

[SimonUnge]

OK, I'll ping him when he returns! Thank you!

[mkuratczyk]

No objection from me. I don't know the details of some of the other peer discovery plugins so I'm not 100% sure it'd be a good idea to expose it as a generic option but perhaps yes.

[michaelklishin]

I don't think there would be any mechanism-specific issues. We target the case where a node could not join the cluster to begin with after all :)

[SimonUnge]

Yeah, and as an 'opt-in' option only... So, I'll go ahead and add the schema version of it and see how that PR sits with you later!