Make TES task scheduling activities more scalable

[BMurri]

Problem:
The task update loop (currently the single largest workload TES does) scales in an exponential manner (based on measurements).

Solution:
The task update loop should scale on a linear scale. See below for suggested solution.

Describe alternatives you've considered
See microsoft/CromwellOnAzure#497. This issue will replace that issue.

Additional context
Currently, the Scheduler service passes each active TesTask to BatchScheduler, which performs various API calls to determine the state of that task in Batch, which it uses to then perform operations on behalf of and/or alter the state of the task. This is performed serially.

The proposed solution is instead to perform the following operations in the following order (assignment of responsibilities TBD):

1. In Autopool mode, query all jobs and filter by TesTaskIds, or in auto-scale mode, query all pools & jobs filtering by known pool ids, capturing all task and compute node states at that time. This will drastically reduce the number of Batch API calls when there are large numbers of active TesTasks, while not overloading the system when there are very few, by taking advantage of using Lists to query Batch state. This is the part that causes the most load in TES when scaling up workloads.
2. For each TesTask, process the retrieved information from step 1 and run each state/task pairing that through the current state machine, except that any operations that can be combined across tasks will be combined (to reduce Azure API loads).
3. Update all tasks in the DB.

Examples of 2. include creating pools/jobs (in auto-scale mode, since they are sharable), retrieving image container credentials, etc. The means of combining operations is TBD.

[ngambani]

@BMurri is this fixed with the most recent PR #442 ?

[BMurri]

@ngambani We'll know once I've scale tested it.

[BMurri]

With the node runner, we now have events