⚙ Streamline github actions workflows

[MatMoore]

Purpose

We have reviewed our workflow and environment usage; see miro diagram: https://miro.com/app/board/uXjVK8MOx_c=/

Now we are in beta and are about to onboard our first users, this is a good time to make sure our github actions workflows

• supports our desired workflow and doesn't get in the way of it
• ensure code passes through all the quality gates before it can reach production

We have decided we can deploy automatically to test, without manual approval, but we should have manual approval before deploying to production (and possibly pre-production).

Before we hit approve we want to notify users and stakeholders of what's changed. We also intend to create Github releases prior to rolling out to live, in order to

1. track compatibility with datahub versions
2. allow our frontend releases to be consumed by other government departments.

Proposal

• Rename existing workflows and remove unused ones
  • test (test-and-deploy.yml) -> "Test and deploy branch to dev"
  • test (test-datahub-client.yml)-> Remove, as long as the tests get run in the above workflow
• Extend the CI/CD workflow that runs on main (.github/workflows/deploy.yml)
  • Auto-deploy to Test (if unit tests pass - make this
  • Gated deploys to preprod and prod
  • Trigger this workflow when a tag/release is created, rather than merge to main
• "Release Datahub Client library" -> Merge with .github/workflows/deploy.yml
• Keep GHA running tests on each push but try to solve "hanging open actions" issue: jobs left in "waiting" state if we choose not to deploy to dev (this creates a lot of notification spam)

Acceptance criteria

• One workflow_dispatch action to test and deploy manually from any branch into dev
• One CI/CD workflow, triggered by the creation of a release, which runs tests, automatically builds and deploys to test, and pushes the library to PyPI. Steps and approvals for subsequent deploys into preprod and prod.
• Every workflow has a unique name

[MatMoore]

Example of jobs left in a waiting state: https://github.com/ministryofjustice/find-moj-data/actions/workflows/test-and-deploy.yml

Github treats the deploy to dev as a thing that has to happen, but actually it is optional, and we don't need to do it for every single push.

• Can these actions be invalidated by later jobs on the same branch?
• Should we take this out of the workflow entirely and just have a workflow_action trigger for dev?

[MatMoore]

Should deployment be triggered by creating a github release, instead of merge to main? This syncs up the deployment to MOJ with the release & versioning process.