Topology change resilience for incremental repair

[Miles-Garnsey]

Ensure that, in an incremental repair, the replica list is updated on every repair run. This addresses cases where the node IPs change.

Fixes #1213

[Miles-Garnsey]

This is still a work in progress. I have written a test (which is passing), but it isn't clear to me that this should actually be working.

1. On line 470 I make a call to clusterFacade.getRangeToEndpointMap without actually having called .connect.
2. When attempting to extract the token range from segment, I only extract the first and last tokens. I need to confirm that this will work even when vnodes are present (although given my limited understanding of what a segment is, I think it may be OK?)

[Miles-Garnsey]

Manual testing of this change suggests that there are possible improvements still available, but that it does address the specific requirements in the ticket.

First, I tried starting the repair, pausing it, then doing a rolling restart on the cluster. I've included a log below showing the results. The problem there was that no coordinators could be found. test-dc1-reaper-94769775c-npd4n.log

But that problem is subtly different to the one we are trying to address here. The matter at hand is not dealing with a full rolling restart (where every IP changes) but just the case where one node's IP changes. I recommend creating another ticket to handle this more extreme case by redoing the DNS lookup so that the list of potential coordinators is refreshed for cases where an FQDN is used as the contact point.

So to make this test more specific I created a new repair, paused it, took note of the host which the current segment was running against, and restarted just that node (leaving the other two in place).

I do still get the below error, but the repair appears to reschedule correctly:

java.lang.IndexOutOfBoundsException: Index 0 out of bounds for length 0
	at java.base/jdk.internal.util.Preconditions.outOfBounds(Preconditions.java:64)
	at java.base/jdk.internal.util.Preconditions.outOfBoundsCheckIndex(Preconditions.java:70)
	at java.base/jdk.internal.util.Preconditions.checkIndex(Preconditions.java:248)
	at java.base/java.util.Objects.checkIndex(Objects.java:372)
	at java.base/java.util.ArrayList.get(ArrayList.java:459)
	at io.cassandrareaper.service.RepairRunner.isAllowedToRun(RepairRunner.java:267)
	at io.cassandrareaper.service.RepairRunner.run(RepairRunner.java:234)
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
	at com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:125)
	at com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:57)
	at com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:78)
	at com.codahale.metrics.InstrumentedScheduledExecutorService$InstrumentedRunnable.run(InstrumentedScheduledExecutorService.java:241)
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at com.codahale.metrics.InstrumentedThreadFactory$InstrumentedRunnable.run(InstrumentedThreadFactory.java:66)
	at java.base/java.lang.Thread.run(Thread.java:829)

You'll note from the below screenshot that it does switch to a different coordinator for the second two segments, which I think is the behaviour we want.

[Miles-Garnsey]

We took this back to the drawing board because our JMX calls were not returning any way to identify the primary replica when doing range -> endpoint queries.

This new approach instead tracks the hostID associated with a given segment and looks up the current IP associated with it when starting each run. It then uses that IP as the coordinator to ensure we are hitting the same endpoint even if the IP has changed.

[adejanovski]

Awesome sauce @Miles-Garnsey! Approved ✅