Ability to run all tests in parallel

[binarylogic]

WARNING: Github has asked up explicitly not to spin up a lot of parallel jobs on large instances. This task shold use regular/small instances.

It looks like Github Actions has the ability to run jobs as a matrix (do not use large instances!). I would like to run every test case in parallel unless otherwise specified. For example:

• PR comment /test runs all tests in parallel.
• PR comment /test -t docker_partial_events_merging_correctness run a single test.
• PR comment /test -t docker_partial_events_merging_correctness -t disk_buffer_performance runs only the tests specified.
• PR comment /test -t *_performance runs all tests ending with _performance.

All tests are designed to be namespaced, there should be no problem running them all in parallel. I am also not concerned about resource usage or cost. If we bump into AWS limits we can request an increase.

[binarylogic]

I added a fourth requirement, which will allow us to run specific tests only. This, if I had to guess, will be how we run tests most of the time.

[MOZGIII]

So, what the /test command do besides running tests? Should it just run tests and post a link to the execution log? Or should it also run bin/compare and post the results?

[binarylogic]

Yep, have it run bin/compare for now. I plan to work on the output. The way you have it should make it very easy for me to customize.

[Hoverbear]

So you'd want a machine running each of https://github.com/timberio/vector-test-harness/tree/master/cases individually? Is it because these are quite long running tests?

[MOZGIII]

Test harness uses very few resources from the host on where it's executing.
However, prior to invoking the test harness, we need to build vector and pack it as a deb. This is the only resource hungry part.

I think technically using build matrices is possible here, if we decouple vector building and test harness invocation - test harness itself doesn't create a lot of load either on the CPU nor on I/O side (on the host).
I do remember Github people asking us not to create a lot of load, and I think if we run test harness in a matrix with a vector pre-built in previous steps - it would work for us and not hurt our neighbors too much. I don't actually know how the resources are scheduled and reserved, so maybe that's a bad idea.

We can alternatively implement parallel execution at the test harness level, rather than at the test harness action level - it'll give us even more flexibility, and overall should be a better option really.

[Hoverbear]

I don't actually know how the resources are scheduled and reserved, so maybe that's a bad idea.

We're sharing a XL worker pool of about 80 with the Rust project right now, and when we talked to Joe we said we'd be doing around 100-200 jobs a day.

We can alternatively implement parallel execution at the test harness level, rather than at the test harness action level - it'll give us even more flexibility, and overall should be a better option really.

Agreed, I note we asked them for the XL runners on Vector itself, so we could possibly run the build step there and then have the harness go do its thing on small runners.

[blt]

This ticket's concerns are superseded by our soak testing infrastructure. Parallel running is accomplished there.