Github Actions integration

[MOZGIII]

Design plan so far:

1. Trigger a Github Action on issue comment.

2. Check the context:

   • comment body must be in a form of /test command
   • comment must be authored by a user with a write access to a repo
   • comment must be on a Pull Request (not on a regular Issue)

   Abort if any of the above are not fulfilled.

3. Clone the PR branch and build vector from it.

   This needs verification since there's a concern that standard Github Action runner might not have
   sufficient resources to build vector.
   Note we can get the Pull Request info from the event file at $GITHUB_EVENT_PATH.

   An alternative approach is building the vector at the test harness VM. It might be easier too, as
   we'll be able to perform the standard cargo install from git:

   cargo install --git https://github.com/timberio/vector --rev "<PR_COMMIT_SHA>"

   Note that to determine the commit hash, we'll have to perform some Github API calls.
   We can avoid doing that if we just use the PR branch: pull/<PR_ID>/head.

   cargo install --git https://github.com/timberio/vector --branch "pull/<PR_ID>/head"

4. Invoke the vector-test-harness docker image.

   We'll have to add support for using the vector executable built on a previous step - probably
   add a way to upload it into the test VM and replace the preinstalled vector instance.

   The command sequence we'll need to run is something like this:

   • bin/test -t some_test
   • bin/compare -t some_test > "$GITHUB_WORKSPACE/output"

   We place the whatever output we want to respond with at $GITHUB_WORKSPACE/output.

5. Finally, we take the contents of $GITHUB_WORKSPACE/output and post it as a comment to the PR. We should also provide a link to the Github Action execution in the said comment, so additional context is easily available at hand.

Failure handling:

In general, we should never fail the action, cause it'll send a failure email to whoever did the latest commit to master.

• If the command is not recognized in the comment - terminate the action without failure so that people don't get an email about failed action on every comment. Ideally, do not even run the action. Or remove the execution log so it doesn't clutter the Actions tab.
• If the user is not authorized to run the action - post a comment about it in response to the command invocation comment. Do not fail the action job, just terminate it with success or other neutral state (if any).
• If the deb build or test harness failed - post a comment about it in response to the user comment with the link to the action execution log - ideally to the exact location where the failure occurred. Do not fail the task (otherwise, the user will get two emails - one for a comment we post in response, and one for the action failure).

[binarylogic]

This sounds good to me! In regards to building Vector, I would check out these docs (select the "Docker" tab). We have Docker images available for building Vector which should simplify that process.

[MOZGIII]

I was looking more into our build and packaging system, and I figured we can just build the .deb just as we go for the publishing - via the make package-deb. This will enable reusing the current way vector is managed within the test harness (it's installed from the .deb package). This is great, cause it solves the problem of carrying the systemd units around - those can be quite important for the test harness.
It's still unclear whether we want to do that at the Github Actions side or at the test harness VM - I'd try doing it at Github Actions end first.

[MOZGIII]

If someone's interested, the experiments are underway at https://github.com/timberio/vector-test-harness-github-actions-test-repo

[binarylogic]

Nice! I assume the changes here will be merged into the test harness itself?

[MOZGIII]

I'm still thinking about how to split the logic and where to put the interface boundary. Most likely the code from here will be moved to the vector repo, and some of it I might extract into a separate action and put either at test harness repo, or into a separate repo. Or maybe even keep at vector repo as well.

[Hoverbear]

We've been in contact with some folks from Github about larger/custom runners. :) Will report back when we hear from them.

[MOZGIII]

I'm at the point where test run finally passes as I want it!
Please try leaving a /test comment at vectordotdev/vector-test-harness-github-actions-test-repo#3
There are still a lot of potential issues, for instance, concurrent runs will most likely break things, but overall - it works!

[binarylogic]

Very cool! This is a great start. I want to play around a little more, but this looks to be simple enough to extend (accept /test arguments, run tests in parallel, etc). Really digging the Github actions integration too.

[MOZGIII]

Yeah, I've had a lot of trouble with the Github Actions, eventually ended up looking at their runner code here. One thing it particular is the env var initialization and expression evaluation order are intertwined, and different for one action type to another (docker action vs node action). I've had other minor surprises too.
All in all, it's not as good as I expected it to be, but it's decent, and imo, Github Actions are still better than Circle CI.

[MOZGIII]

There are a few things to polish before we merge it in:

• we need to check whether the user has write access to the repo
• switch the action status from failing to canceling if it's a comment on a regular issue or the access check from the previous step fails
• switch the test harness to run on a dedicated AWS account (Separate AWS account #16)

Everything else can be addressed later.

[blt]

This ticket's concerns are superseded by our soak testing infrastructure. Any future iteration of test-harness work will use the basic infrastructure of that work.