Zarf Agent mutating webhook (zarf-dev#306)

- Introduce mutating webhook (zarf agent)
- Mutate podSpec image paths 
- Mutate podSpec imagePullSecret
- Mutate Flux GitRepository URL
- Mutate Flux GitRepository secret

- Add dev makefile targets for image rebuilds
- Add manual github action for building a new zarf agent image

- Update Big Bang example package 1.28 -> 1.33
- Migrate EFK -> PLG for Big Bang example
- Fix OPA Gatekeeper failures on KIND clusters

- Add Zarf Agent ADR

Co-authored-by: Jon Perry <yrrepnoj@gmail.com>

Author: jeff-mccoy

.github/workflows/build-rust-injector.yml

@@ -26,7 +26,6 @@ jobs:
       - name: "Install cosign"
         uses: sigstore/cosign-installer@v2.1.0
 
-
       - name: "Install Rust And Build"
         uses: gmiam/rust-musl-action@v1.1.1
         with:

.github/workflows/build-zarf-agent.yml

@@ -12,29 +12,50 @@ on:
         default: "master"
 
 jobs:
-  build-injector:
+  build-agent:
     runs-on: ubuntu-latest
     steps:
+      - name: "Install GoLang"
+        uses: actions/setup-go@v2
+        with:
+          go-version: 1.18.x
+
       - name: "Checkout Repo"
         uses: actions/checkout@v2
         with:
           ref: ${{ github.event.inputs.branchName }}
+      
+      - name: "Setup caching"
+        uses: actions/cache@v3
+        with:
+          path: |
+            ~/.cache/go-build
+            ~/go/pkg/mod
+          key: ${{ runner.os }}-go-${{ hashFiles('**/go.sum') }}
+          restore-keys: |
+            ${{ runner.os }}-go-
 
       - name: "Install cosign"
         uses: sigstore/cosign-installer@v2.1.0
 
-      - name: Set up Docker Buildx
+      - name: "Set up Docker Buildx"
         id: buildx
         uses: docker/setup-buildx-action@v2
 
-      - name: Login to Docker Hub
+      - name: "Login to Docker Hub"
         uses: docker/login-action@v1
         with:
           username: ${{ secrets.DOCKERHUB_USERNAME }}
           password: ${{ secrets.DOCKERHUB_TOKEN }}
 
+      - name: "Build zarf binaries"
+        run: make build-cli-linux
+
+      - name: "Rename binaries for packaging"
+        run: mv build/zarf build/zarf-linux-amd64 && mv build/zarf-arm build/zarf-linux-arm64
+
       - name: "Build and Publish the Image"
-        run: buildx build --push --platform linux/arm64/v8,linux/amd64 --tag defenseunicorns/zarf-agent:${{ github.event.inputs.versionTag }} .
+        run: docker buildx build --push --platform linux/arm64/v8,linux/amd64 --tag defenseunicorns/zarf-agent:${{ github.event.inputs.versionTag }} .
 
       - name: "Sign the Image"
         run: cosign sign --key awskms:///${{ secrets.COSIGN_AWS_KMS_KEY }} -a release-engineer=https://github.com/${{ github.actor }} -a version=${{ github.event.inputs.versionTag }} defenseunicorns/zarf-agent:${{ github.event.inputs.versionTag }}

.github/workflows/release.yml

@@ -13,18 +13,33 @@ jobs:
         uses: actions/setup-go@v2
         with:
           go-version: 1.18.x
+      
       - name: Checkout Repo
         uses: actions/checkout@v2
+      
+      - name: "Setup caching"
+        uses: actions/cache@v3
+        with:
+          path: |
+            ~/.cache/go-build
+            ~/go/pkg/mod
+          key: ${{ runner.os }}-go-${{ hashFiles('**/go.sum') }}
+          restore-keys: |
+            ${{ runner.os }}-go-        
+     
       - name: Build Things For Release
         run: make build-cli init-package
+      
       - name: Set AWS Credentials
         uses: aws-actions/configure-aws-credentials@v1
         with:
           aws-access-key-id: ${{ secrets.AWS_GOV_ACCESS_KEY_ID }}
           aws-secret-access-key: ${{ secrets.AWS_GOV_SECRET_ACCESS_KEY }}
           aws-region: us-gov-west-1
+      
       - name: Push Release Artifacts to S3 Bucket
         run: aws s3 cp build s3://zarf-public/release/${{ github.ref_name }} --region us-gov-west-1 --recursive
+      
       - name: Create a Release For This Tag
         uses: softprops/action-gh-release@v1
         if: startsWith(github.ref, 'refs/tags/')

.github/workflows/test-k3d.yml

@@ -10,12 +10,26 @@ jobs:
         uses: actions/setup-go@v2
         with:
           go-version: 1.18.x
+
       - name: "Checkout Repo"
         uses: actions/checkout@v2
+
+      - name: "Setup caching"
+        uses: actions/cache@v3
+        with:
+          path: |
+            ~/.cache/go-build
+            ~/go/pkg/mod
+          key: ${{ runner.os }}-go-${{ hashFiles('**/go.sum') }}
+          restore-keys: |
+            ${{ runner.os }}-go-        
+
       - name: "Build CLI"
         run: make build-cli-linux
+
       - name: "Make Packages"
         run: make init-package package-example-game package-example-data-injection package-example-gitops-data package-example-compose
+
       - name: "Run Tests"
         # NOTE: This test run will create its own K3d cluster. A single cluster will be used throughout the test run.
         run: TESTDISTRO=k3d make test-e2e

.github/workflows/test-k3s.yml

@@ -10,12 +10,26 @@ jobs:
         uses: actions/setup-go@v2
         with:
           go-version: 1.18.x
+      
       - name: "Checkout Repo"
         uses: actions/checkout@v2
+      
+      - name: "Setup caching"
+        uses: actions/cache@v3
+        with:
+          path: |
+            ~/.cache/go-build
+            ~/go/pkg/mod
+          key: ${{ runner.os }}-go-${{ hashFiles('**/go.sum') }}
+          restore-keys: |
+            ${{ runner.os }}-go-
+
       - name: "Build CLI"
         run: make build-cli-linux
+      
       - name: "Make Packages"
         run: make init-package package-example-game package-example-data-injection package-example-gitops-data package-example-compose
+      
       - name: "Run Tests"
         # NOTE: "PATH=$PATH" preserves the default user $PATH. This is needed to maintain the version of go installed
         #       in a previous step. This test run will use Zarf to create a K3s cluster, and a brand new cluster will be

.github/workflows/test-kind.yml

@@ -10,14 +10,29 @@ jobs:
         uses: actions/setup-go@v2
         with:
           go-version: 1.18.x
+      
       - name: "Checkout Repo"
         uses: actions/checkout@v2
+      
       - name: "Create k8s Kind Cluster"
         uses: helm/kind-action@v1.2.0
+      
+      - name: "Setup caching"
+        uses: actions/cache@v3
+        with:
+          path: |
+            ~/.cache/go-build
+            ~/go/pkg/mod
+          key: ${{ runner.os }}-go-${{ hashFiles('**/go.sum') }}
+          restore-keys: |
+            ${{ runner.os }}-go-
+
       - name: "Build CLI"
         run: make build-cli-linux
+      
       - name: "Make Packages"
         run: make init-package package-example-game package-example-data-injection package-example-gitops-data package-example-compose
+      
       - name: "Run Tests"
         # NOTE: We want to test providing a cluster to the test framework so this one creates its own KinD cluster
         #       rather than having the test suite do it. The K3d tests do a self-provisioned cluster and the K3s tests

.gitignore

@@ -22,6 +22,7 @@ test/tf/public-ec2-instance/.terraform
 terraform.tfstate
 terraform.tfstate.backup
 .terraform.lock.hcl
+cosign.key
 .zarf*
 zarf-pki
 .scratch/

Dockerfile

@@ -0,0 +1,9 @@
+FROM scratch
+ARG TARGETARCH
+
+ADD "build/zarf-linux-$TARGETARCH" /zarf
+EXPOSE 8443
+
+ENV USER=zarf
+
+CMD ["/zarf", "agent", "-l=trace"]

Makefile

@@ -16,6 +16,8 @@ ifneq ($(UNAME_S),Linux)
 	endif
 endif
 
+CLI_VERSION := $(if $(shell git describe --tags), $(shell git describe --tags), "UnknownVersion")
+BUILD_ARGS := -s -w -X 'github.com/defenseunicorns/zarf/src/config.CLIVersion=$(CLI_VERSION)'
 .DEFAULT_GOAL := help
 
 .PHONY: help
@@ -38,17 +40,35 @@ vm-destroy: ## Destroy the VM
 clean: ## Clean the build dir
 	rm -rf build
 
-build-cli-linux: build-injector-registry ## Build the Linux CLI
-	cd src && $(MAKE) build
+build-cli-linux-amd: build-injector-registry
+	CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build -ldflags="$(BUILD_ARGS)" -o build/zarf src/main.go
 
-build-cli-mac: build-injector-registry ## Build the Mac CLI
-	cd src && $(MAKE) build-mac
+build-cli-linux-arm: build-injector-registry
+	CGO_ENABLED=0 GOOS=linux GOARCH=arm64 go build -ldflags="$(BUILD_ARGS)" -o build/zarf-arm src/main.go
 
-build-cli: build-injector-registry build-cli-linux build-cli-mac ## Build the CLI
+build-cli-mac-intel: build-injector-registry
+	GOOS=darwin GOARCH=amd64 go build -ldflags="$(BUILD_ARGS)" -o build/zarf-mac-intel src/main.go
+
+build-cli-mac-apple: build-injector-registry
+	GOOS=darwin GOARCH=arm64 go build -ldflags="$(BUILD_ARGS)" -o build/zarf-mac-apple src/main.go
+
+build-cli-linux: build-cli-linux-amd build-cli-linux-arm 
+
+build-cli: build-cli-linux-amd build-cli-linux-arm build-cli-mac-intel build-cli-mac-apple ## Build the CLI
 
 build-injector-registry:
 	cd src/injector/stage2 && $(MAKE) build-bootstrap-registry
 
+# Inject and deploy a new dev version of zarf agent for testing (should have an existing zarf agent deployemt)
+# @todo: find a clean way to support Kind or k3d: k3d image import $(tag)
+dev-agent-image:
+	$(eval tag := defenseunicorns/dev-zarf-agent:$(shell date +%s))
+	$(eval arch := $(shell uname -m))
+	CGO_ENABLED=0 GOOS=linux go build -o build/zarf-linux-$(arch) src/main.go
+	DOCKER_BUILDKIT=1 docker build --tag $(tag) --build-arg TARGETARCH=$(arch) . && \
+	kind load docker-image zarf-agent:$(tag) && \
+	kubectl -n zarf set image deployment/agent-hook server=$(tag)
+
 init-package: ## Create the zarf init package, macos "brew install coreutils" first
 	$(ZARF_BIN) package create --confirm --architecture amd64
 	$(ZARF_BIN) package create --confirm --architecture arm64

README.md

@@ -287,7 +287,7 @@ Zarf is written entirely in [go](https://go.dev/), except for a single 400Kb bin
 - The OCI Registries used are both from [Docker](https://github.com/distribution/distribution)
 - Currently, the Registry and Git servers _are not HA_, see [#375](https://github.com/defenseunicorns/zarf/issues/376) and [#376](https://github.com/defenseunicorns/zarf/issues/376) for discussion on this
 - To avoid TLS issues, Zarf binds to `127.0.0.1:31999` on each node as a [NodePort](https://kubernetes.io/docs/concepts/services-networking/service/#type-nodeport) to allow all nodes to access the pod(s) in the cluster
-- Until [#306](https://github.com/defenseunicorns/zarf/pull/306) is merged, during helm install/upgrade a [Helm PostRender](https://helm.sh/docs/topics/advanced/#post-rendering) function is called to mutate images and [ImagePullSecrets](https://kubernetes.io/docs/concepts/containers/images/#specifying-imagepullsecrets-on-a-pod) so the deployed resources use the NodePort binding
+- A [mutating webhook](https://kubernetes.io/docs/reference/access-authn-authz/extensible-admission-controllers/) we call the Zarf Agent handles updating pod's [ImagePullSecrets](https://kubernetes.io/docs/concepts/containers/images/#specifying-imagepullsecrets-on-a-pod) so the deployed resources use the NodePort binding
 - Zarf uses a custom injector system to bootstrap a new cluster. See the PR [#329](https://github.com/defenseunicorns/zarf/pull/329) and [ADR](docs/adr/0003-image-injection-into-remote-clusters-without-native-support.md) for more details on how we came to this solution.  The general steps are listed below:
   - Get a list of images in the cluster
   - Attempt to create an ephemeral pod using an image from the list
@@ -301,5 +301,4 @@ Zarf is written entirely in [go](https://go.dev/), except for a single 400Kb bin
 ### Zarf Architecture
 ![Architecture Diagram](./docs/architecture.drawio.svg)
 
-
 [Source DrawIO](docs/architecture.drawio.svg)

docs/adr/0005-mutating-webhook.md

@@ -0,0 +1,21 @@
+# 5. Mutating Webhook
+
+Date: 2022-05-18
+
+## Status
+
+Accepted
+
+## Context
+
+Currently Zarf leverages [Helm Post Rendering](https://helm.sh/docs/topics/advanced/#post-rendering) to mutate image paths and secrets for K8s to use the internal [Zarf Registry](../../packages/zarf-registry/README.md). This works well for simple K8s deployments where Zarf is performing the actual manifest apply but fails when using a secondary gitops tools suchs as [Flux](https://github.com/fluxcd/flux2), [ArgoCD](https://argo-cd.readthedocs.io/en/stable/), etc. At that point, Zarf is unable to provide mutation and it is dependent on the package author to do the mutations themselves using rudimentary templating. Further, this issue also exists when for CRDs that references the [git server](../../packages/gitea/README.md). A `zarf prepare` command was added previously to make this less painful, but it still requires additional burden on package authors to do something we are able to prescribe in code. 
+
+## Decision
+
+A [mutating webhook](https://kubernetes.io/docs/reference/access-authn-authz/extensible-admission-controllers/) is standard practice in K8s and there [are a lot of them](https://kubernetes.io/docs/reference/access-authn-authz/admission-controllers/#what-does-each-admission-controller-do). Using the normal Zarf componenet structure and deployment strategy we can leverage a mutating webhook to perform automatic imagePullSecret binding and image path updates as well as add additional as-needed mutations such as updating the [GitRepository](https://fluxcd.io/docs/components/source/gitrepositories/) CRD with the appropriate secret and custom URL for the git server if someone is using Flux.
+
+## Consequences
+
+While deploying the webhook will greatly reduce the package development burden, the nature of how helm manages resources still means we will have to be careful how we apply secrets that could collide with secrets deployed by helm with other tools. Additionally, to keep the webhook simple we are foregoing any side-effects in this iteration such as creating secrets on-demand in a namespace as it is created.  Adding side effects carries with it the need to roll those back on failure, handle additional RBAC in the cluster and integrate with the K8s API in the webhook. Therefore, some care will have to be taken for now with how registry and git secrets are generated in a namespace. For example, in the case of [Big Bang](https://repo1.dso.mil/platform-one/big-bang/bigbang) these secrets can be created by those helm charts if we pass in the proper configuration. 
+
+Another benefit of this approach is another layer security for Zarf clusters.  The Zarf Agent will act as an intermediary not allowing images not in the Zarf Registry or git repos not stored in the internal git server.  

docs/architecture.drawio.svg

@@ -18,7 +18,7 @@ Zarf's work necessitates that some components are "always on" (a.k.a. required &
 |                         | Description                                                                                                          |
 | ----------------------- | -------------------------------------------------------------------------------------------------------------------- |
 | container-seed-registry | Adds a container registry so Zarf can bootstrap itself into the cluster.                                             |
-| container-registry      | Adds a container registry service—[docker registry](https://docs.docker.com/registry/)—into the cluster. |
+| zarf-registry      | Adds a container registry service—[docker registry](https://docs.docker.com/registry/)—into the cluster. |
 
  
 

docs/components.md

@@ -12,7 +12,7 @@ To test create a virtual area to test all examples, you can run `make all` or `m
 
 | Example                                                          |      Description      |
 |------------------------------------------------------------------|-------------|
-| [big-bang](./big-bang/README.md)                                 |  Demo BigBang v1.28.0 with all of its core services |
+| [big-bang](./big-bang/README.md)                                 |  Demo BigBang v1.33.0 with all of its core services |
 | [composable-packages](./composable-packages/README.md)           |  Demo building packages using components from other packages   |
 | [data-injection](./data-injection/README.md)                     |  Demo injecting data into a pod running on cluster  |
 | [game](./game/README.md)                                         |  Demo deploying old-school DOS games |

examples/README.md

@@ -1,24 +1,9 @@
 bases:
-  - git::https://repo1.dso.mil/platform-one/big-bang/bigbang.git/base?ref=tags/1.28.0
+  - git::https://repo1.dso.mil/platform-one/big-bang/bigbang.git/base?ref=tags/1.33.0
 
 configMapGenerator:
   - name: common
     namespace: bigbang
     behavior: merge
     files:
       - values.yaml
-
-resources:
-  - git-secret.yaml
-
-patchesStrategicMerge:
-  - |-
-    apiVersion: source.toolkit.fluxcd.io/v1beta1
-    kind: GitRepository
-    metadata:
-      name: bigbang
-      namespace: bigbang
-    spec:
-      url: http://zarf-gitea-http.zarf.svc.cluster.local:3000/zarf-git-user/mirror__repo1.dso.mil__platform-one__big-bang__bigbang.git
-      secretRef:
-        name: zarf-git-secret