Let minions try to connect to master indefinitly

[dwoz]

What does this PR do?

Do not exit when the minion can't connect to a master.

What issues does this PR fix or reference?

#50854

Tests written?

No

Commits signed with GPG?

Yes

[dwoz]

@Ch3LL @DmitryKuzmenko @garethgreenaway

I've tested that this also resolves #50791. Pun intended ;)

Scenario's tested:

• Single master not started. The minion connects when master is started.
• Multiple masters with 1 not being able to resolve. The minion fails over to second master.
• Multiple master with 1 not started. The minion fails over to the second master.
• Multiple masters with none started. The minion connects a master is started.

[damon-atkins]

Random sleep time adjustment? i.e. to prevent all of the minions trying at about the same time to connect, and causing a wave effect against the master when it returns. It not I suggest its added to the code. + or - 10% of the interval time. interval - (interval * 0.05) + random(interval* 0.1)

[Ch3LL]

@dwoz i believe when @garethgreenaway was trying to fix this initially it was to fix multi-master with dns (dns names as the master in the minion config). Did you happen to test with dns or ip addresses?

[Ch3LL]

nevermind. I looked it up here: #49764 and it seems it should only work when using the new retry_dns_count configuratino

[dwoz]

@Ch3LL I my testing I used DNS with resolvable and un-resolvable hosts.

@damon-atkins That is a good idea but this is a bugfix. Changes should go in a feature.

[cachedout]

@damon-atkins That is a good idea but this is a bugfix. Changes should go in a feature.

Given that this is still pre-release, this might still be feasible, IMHO.

[dwoz]

@damon-atkins That is a good idea but this is a bugfix. Changes should go in a feature.

Given that this is still pre-release, this might still be feasible, IMHO.

I'd still like to see that go in a different PR to keep added functionality separate from the bugfix found here.

[cachedout]

We need to ensure this does not regress this fix, IMHO:

e66dc18

[dwoz]

@cachedout We discussed this on slack and @garethgreenaway agreed that this change is what we want. Just waiting for @garethgreenaway to sign off by approving this PR.