Investigate and fix test failures

[Zruty0]

Our commit success rate is below 50% at this point.

We no longer have the annoying deadlock plaguing the builds, but somehow the builds are still failing quite often. Let's spend some time and improve that story.

From casual look, there is no single culprit: we seem to randomly fail to initialize build environment, and random tests seem to fail in random places. I guess this may have to do with resource-constrained runs: maybe we should actually try running xUnit tests sequentially and see if the sporadic failures go away?

[Ivanidzo4ka]

Should we at least create list of failing tests?
Let me start it:
MulticlassTreeFeaturizedLRTest - MacOS - Debug
System.InvalidOperationException : MTA thread failed
---- Assert.InRange() Failure
Range: (74.9925 - 75.0075)
Actual: 72

MultithreadedPrediction - MacOS - Debug
at Microsoft.ML.Runtime.Contracts.CheckDecode(IExceptionContext ctx, Boolean f) in Microsoft.ML.Core/Utilities/Contracts.cs:line 713
at Microsoft.ML.Runtime.Model.ModelLoadContext.LoadNonEmptyString() in Microsoft.ML.Data/Model/ModelLoadContext.cs:line 163
at Microsoft.ML.Runtime.Data.TermTransform.TermMap.TextImpl.Create(ModelLoadContext ctx, IExceptionContext ectx) in /Microsoft.ML.Data/Transforms/TermTransformImpl.cs:line 603
at Microsoft.ML.Runtime.Data.TermTransform.TermMap.Load(ModelLoadContext ctx, IHostEnvironment ectx, CodecFactory codecFactory) in /Microsoft.ML.Data/Transforms/TermTransformImpl.cs:line 518

FastTreeBinaryClassificationCategoricalSplitTest MacOS - Debug

at Microsoft.ML.Runtime.RunTests.BaseTestBaseline.Done() in /Users/buildagent/agent/_work/1/s/test/Microsoft.ML.TestFramework/BaseTestBaseline.cs:line 149
at Microsoft.ML.Runtime.RunTests.TestPredictors.FastTreeBinaryClassificationCategoricalSplitTest() in /Users/buildagent/agent/_work/1/s/test/Microsoft.ML.Predictor.Tests/TestPredictors.cs:line 725

TrainAndPredictIrisModelWithStringLabelTest MacOS - Debug
Assert failed: longIdx=133, invariants.Length=133
Expected: True
Actual: False
at Microsoft.ML.Runtime.Learners.SdcaTrainerBase2.TrainCore(IChannel ch, RoleMappedData data, LinearPredictor predictor, Int32 weightSetCount) in /src/Microsoft.ML.StandardLearners/Standard/LinearClassificationTrainer.cs:line 517 at Microsoft.ML.Runtime.Learners.StochasticTrainerBase2.TrainModelCore(TrainContext context) in /src/Microsoft.ML.StandardLearners/Standard/StochasticTrainerBase.cs:line 42

[Zruty0]

@montebhoover , would you mind taking a look?

[frank-dong-ms-zz]

These issues are fixed recently, some race condition issues.