Injector with external Vault server?

[smurfralf]

The scenario we want to support is to use a vault server which pre-exists the kubernetes cluster. We want the vault-k8s injector capability to talk to this vault server.

Is this scenario supported? I see that the injector command (vault-k8s/subcommand/injector/flags.go) takes a vault-address argument. And I see that deployment (vault-helm/templates/injector-deployment.yaml) sets the AGENT_INJECT_VAULT_ADDR env variable.

If the injector can be configured to use an external service, please document an example configuration that also disables installing vault as a service inside kubernetes.

[jasonodonnell]

@smurfralf This scenario should work, provided pods can communicate with the external Vault cluster. You will need to set AGENT_INJECT_VAULT_ADDR on the deployment, as you already noticed. I will document the manual installation process using the deploy files in this repo.

[ls-brentsmith]

I'm struggling with using the deploy manifests from this repo, which i'm attempting to use because the helm charts are currently not working as intended with regards to global.enable variable, among other templating issues.

FWIW - I was able to get it working by changing the following

1) I added the following rbac configuration which is reused from the kubernetes auth documentation, but it's not clear to me why, as it seems the admission registration should cover it.
2) I changed AGENT_INJECT_VAULT_ADDR to my vault server url
3) I needed to make sure the vault-injector service account existed in the same namespace as my application, which makes sense in retrospect, but wasn't obvious to me in the manifests or the charts (nothing is ever obvious in a helm chart to be fair :) )

I will report back here as I become less confused :)

[jasonodonnell]

Hi @ls-brentsmith,

I added the following rbac configuration which is reused from the kubernetes auth documentation, but it's not clear to me why, as it seems the admission registration should cover it.

That RBAC is not needed by the injector. The injector doesn't actually communicate with Vault at all. The rbac you listed is what Vault uses when authenticating service accounts in Kubernetes.

I changed AGENT_INJECT_VAULT_ADDR to my vault server url

This should really be the only thing you need to do.

I needed to make sure the vault-injector service account existed in the same namespace as my application, which makes sense in retrospect, but wasn't obvious to me in the manifests or the charts (nothing is ever obvious in a helm chart to be fair :) )

The deploy examples in this repository use the vault namespace. You should only need to create a namespace vault and run the following (assuming you configured the AGENT_INJECT_VAULT_ADDR with your external Vault server:

kubectl create namespace vault
kubectl create -f ./deploy/injector-rbac.yaml --namespace=vault
kubectl create -f ./deploy/injector-service.yaml --namespace=vault
kubectl create -f ./deploy/injector-mutating-webhook.yaml --namespace=vault
kubectl create -f ./deploy/deployment.yaml --namespace=vault
kubectl get pods --namespace=vault

[ls-brentsmith]

@jasonodonnell thanks for the quick response.
In the light of morning I saw why I stumbled there.

Follow up question, is it possible to inject a custom auth path for the vault-agents?

[jasonodonnell]

@ls-brentsmith, there's no annotation currently that lets you alter the auth path, however, you can mount your own Vault Agent configuration files using a configmap. Here's an example: https://www.vaultproject.io/docs/platform/k8s/injector/examples.html#configmap-example. Mount path would go in the method section: https://www.vaultproject.io/docs/agent/autoauth/index.html#mount_path

Hope that helps!

[ls-brentsmith]

Thank you for the quick response!
I was just about to come comment the same thing.

[buckner]

Are there plans to add an annotation for custom auth paths? It'd be nice to have a choice between a configmap or just annotations.

[jasonodonnell]

Yes, I'll be adding one @buckner!

[smurfralf]

@jasonodonnell - back to the original topic of this issue...
Your comment WRT setting external URL for AGENT_INJECT_VAULT_ADDR was: "This should really be the only thing you need to do." I did it using name "vault-inj" for the helm install, and the injector is indeed functional. Yay!
But my k8s cluster also has a zombie pod vault-inj-0 showing
Readiness probe failed
because of course this vault server hasn't been initialized.
When I made a stab at setting helm value service.enabled: false then that breaks the injector where logs show
2019-12-23T16:40:45.791Z [INFO] handler: Starting handler.. Listening on ":8080"... Updated certificate bundle received. Updating certs... 2019/12/23 16:40:46 http: TLS handshake error from 10.0.16.98:41948: no certificate available
My guess is that you have dependencies there in your automagic configuration of TLS?

At any rate, solving that isn't my point. What I asked for in the description would be nice:

... please document an example configuration that (allows configuration of an external vault server [preferably without changing the vault-helm template] and that) also disables installing vault as a service inside kubernetes.

[jasonodonnell]

@smurfralf There are some dependencies in the Vault Helm project between the projects, so some stuff will need to change over there for installing the injector standalone.

I'll document the process of configuring the injector with an external Vault service after the new year since HashiCorp is currently in a holiday shutdown.

[VinothChinnadurai]

@jasonodonnell I have a setup of external vault server with consul backend and placed a load balancer in front of vault server in an EKS environment. Now m app has been deployed in separate EKS cluster where I mentioned AGENT_INJECT_VAULT_ADDR as the above mentioned ELB(Port 80). Now my app pod becomes not ready after I injected the init container . STATUS becomes 'Init:0/1' . I think it is something related to token initialization. How can I mention that token here and retrieve the secrets inside the Pod?

[jasonodonnell]

@VinothChinnadurai Did you check the init container logs to see what's happening?

kubectl logs <name of pod> -c vault-agent-init

[VinothChinnadurai]

Yes @jasonodonnell . Please find it here
`URL: PUT http://vault.demo.svc:8200/v1/auth/kubernetes/login
Code: 400. Errors:

• invalid role name "exampleapp-role"" backoff=1.580907647
  2019-12-24T10:09:31.660Z [INFO] auth.handler: authenticating
  2019-12-24T10:09:31.667Z [ERROR] auth.handler: error authenticating: error="Error making API request`

Not sure why it still hits some other service(vault.demo.svc) instead of ELB address

[jasonodonnell]

@VinothChinnadurai Double check the env is set correctly because that looks like the default

vault-k8s/deploy/injector-deployment.yaml

Line 36 in 611492d

value: "https://vault.$(NAMESPACE).svc:8200"

Next retrigger the injection by patching the pod. You'll need to set the status annotation to vault.hashicorp.com/agent-inject-status: update to retrigger the injection (the config needs to be refreshed on the agent).

[VinothChinnadurai]

@jasonodonnell Happy New year! Thanks. It worked for me! But I find an issue with the syncing part where I have updated my secrets through vault API command
curl \ --header "X-Vault-Token: xxx" \ --request POST \ --data @payload.json \ http://myelbaddress:8200/v1/secret/helloworld

Payload.json has
{ "data": { "foo": "bar", "zip": "zap" } }

Post then in the GET command of API, I can receive the updated secrets.
I tried to see the same by logging inside my app pod and verified the file /vault/secret/helloworld
Where it still showed the older values.
Kindly help me with what might be the cause.

[jasonodonnell]

@VinothChinnadurai Lower the TTL of your secret. Updates are based on TTL of the secret.

[trx35479]

@jasonodonnell does this also work with istio service mesh?

[malnick]

@trx35479 We haven't done explicit testing with istio yet, so we can not verify if this will or will not work with it. If you happen to test this, please let us know what you find.

[vispatster]

@VinothChinnadurai Double check the env is set correctly because that looks like the default

vault-k8s/deploy/injector-deployment.yaml

Line 36 in 611492d

value: "https://vault.$(NAMESPACE).svc:8200"

Next retrigger the injection by patching the pod. You'll need to set the status annotation to vault.hashicorp.com/agent-inject-status: update to retrigger the injection (the config needs to be refreshed on the agent).

==========================================================
@jasonodonnell
I have consul+vault installed on one Kubernetes cluster. On the other cluster, I am trying to deploy vault-k8s injector for webhook - https://github.com/hashicorp/vault-k8s.git

Init pod returns the following errors. how can it be fixed? Vault address has been changed to http://vlt.consulvault.172.31.101.63.xip.io

Both the clusters are in the same network and curl command returns the response.

I think I may have to pass the root token(or register secret with Vault) in order to authenticate.

Thank you

curl \
>     -H "X-Vault-Token: token" \
>     -X GET \
>     http://vlt.consulvault.172.31.101.63.xip.io/v1/secret/helloworld
{"request_id":"***","lease_id":"","renewable":false,"lease_duration":2764800,"data":{"password":"foobarbazpass","username":"foobaruser"},"wrap_info":null,"warnings":null,"auth":null}

==> Vault server started! Log data will stream in below:
2020-01-28T18:27:10.042Z [INFO] sink.file: creating file sink
2020-01-28T18:27:10.042Z [INFO] sink.file: file sink configured: path=/home/vault/.token mode=-rw-r-----
==> Vault agent configuration:
Cgo: disabled
Log Level: info
Version: Vault v1.3.1
2020-01-28T18:27:10.042Z [INFO] auth.handler: starting auth handler
2020-01-28T18:27:10.042Z [INFO] auth.handler: authenticating
2020-01-28T18:27:10.042Z [INFO] template.server: starting template server
2020/01/28 18:27:10.042917 [INFO] (runner) creating new runner (dry: false, once: false)
2020/01/28 18:27:10.043280 [INFO] (runner) creating watcher
2020-01-28T18:27:10.043Z [INFO] sink.server: starting sink server
2020-01-28T18:27:10.275Z [ERROR] auth.handler: error authenticating: error="Error making API request.
URL: PUT http://vlt.consulvault.172.31.101.63.xip.io/v1/auth/kubernetes/login
Code: 500. Errors:

* lookup failed: [invalid bearer token, square/go-jose: error in cryptographic primitive]" backoff=2.382848811
2020-01-28T20:21:34.792Z [INFO] auth.handler: authenticating
2020-01-28T20:21:34.806Z [ERROR] auth.handler: error authenticating: error="Error making API request.
URL: PUT http://vlt.consulvault.172.31.101.63.xip.io/v1/auth/kubernetes/login
Code: 500. Errors:
* lookup failed: [invalid bearer token, square/go-jose: error in cryptographic primitive]" backoff=1.805414534
2020-01-28T20:21:36.612Z [INFO] auth.handler: authenticating
2020-01-28T20:21:36.758Z [ERROR] auth.handler: error authenticating: error="Put http://vlt.consulvault.172.31.101.63.xip.io/v1/auth/kubernetes/login: dial tcp: lookup vlt.consulvault.172.31.101.63.xip.io on 10.43.0.10:53: no such host" backoff=1.998397118
2020-01-28T20:21:38.757Z [INFO] auth.handler: authenticating
2020-01-28T20:21:38.761Z [ERROR] auth.handler: error authenticating: error="Put http://vlt.consulvault.172.31.101.63.xip.io/v1/auth/kubernetes/login: dial tcp: lookup vlt.consulvault.172.31.101.63.xip.io on 10.43.0.10:53: no such host" backoff=2.780268301
2020-01-28T20:21:41.541Z [INFO] auth.handler: authenticating

[stevegore]

@jasonodonnell @shlee322 I've noticed the code for setting a custom path for the Kubernetes Auth method has been committed to master. Any chance you guys are going to do a release of that soon?

[jasonodonnell]

Hi @stevegore just released in v0.2.0. Hope that helps!

[dcatalano-figure]

Wrote this for myself and my team. It was pieced together using the awesome HashiCorp docs. All the examples follow what's in hashicorp/vault-guides repo and Learning modules on Hashi site. Hope one of you find this helpful.

Injecting Secrets into Kubernetes Pods via Vault

There are two distinct parts of configuring auto-injection of secrets, Kubernetes & Vault. This will go through how-to configure both in order for auto-injection to work, as well as, a few custom options that should be considered as we deploy to production. This guide combines and refines two separate tutorials provided by Learn Vault for our specific use case.

Kubernetes Auto-Injector Setup (Kubernetes)

Service Accounts

Vault Agent Injector

$ cat sa-vault-agent-injector.yaml
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: vault-agent-injector
  namespace: default
  labels:
    app.kubernetes.io/name: vault-agent-injector
    app.kubernetes.io/instance: vault
  
$ kubectl apply --filename sa-vault-agent-injector.yaml

Vault Auth

$ cat sa-vault-auth.yml
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: vault-auth
  
$ kubectl apply --filename sa-vault-auth.yml

Cluster Role

$ cat clusterrole-vault-agent-injector.yaml
---
# Source: vault/templates/injector-clusterrole.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: vault-agent-injector-clusterrole
  labels:
    app.kubernetes.io/name: vault-agent-injector
    app.kubernetes.io/instance: vault
rules:
- apiGroups: ["admissionregistration.k8s.io"]
  resources: ["mutatingwebhookconfigurations"]
  verbs:
    - "get"
    - "list"
    - "watch"
    - "patch"

$ kubectl apply --filename clusterrole-vault-agent-injector.yaml

Cluster Role Bindings

Vault Agent Injector

$ cat clusterrolebinding-vault-agent-injector.yaml
---
# Source: vault/templates/injector-clusterrolebinding.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: vault-agent-injector-binding
  namespace: default
  labels:
    app.kubernetes.io/name: vault-agent-injector
    app.kubernetes.io/instance: vault
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: vault-agent-injector-clusterrole
subjects:
- kind: ServiceAccount
  name: vault-agent-injector
  namespace: default

$ kubectl apply --filename clusterrolebinding-vault-agent-injector.yaml

Vault Auth

$ cat clusterrolebinding-vault-auth.yml
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
  name: role-tokenreview-binding
  namespace: default
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:auth-delegator
subjects:
- kind: ServiceAccount
  name: vault-auth
  namespace: default
  
$ kubectl apply --filename clusterrolebinding-vault-auth.yml

Service

$ cat svc-vault-agent-injector.yaml
---
# Source: vault/templates/injector-service.yaml
apiVersion: v1
kind: Service
metadata:
  name: vault-agent-injector-svc
  namespace: default
  labels:
    app.kubernetes.io/name: vault-agent-injector
    app.kubernetes.io/instance: vault
spec:
  ports:
  - port: 443
    targetPort: 8080
  selector:
    app.kubernetes.io/name: vault-agent-injector
    app.kubernetes.io/instance: vault
    component: webhook

$ kubectl apply --filename svc-vault-agent-injector.yaml

Mutating Webhook Configuration

$ cat mutatingwebhookconfigurations-vault-agent-injector.yaml
---
# Source: vault/templates/injector-mutating-webhook.yaml
apiVersion: admissionregistration.k8s.io/v1beta1
kind: MutatingWebhookConfiguration
metadata:
  name: vault-agent-injector-cfg
  labels:
    app.kubernetes.io/name: vault-agent-injector
    app.kubernetes.io/instance: vault
webhooks:
  - name: vault.hashicorp.com
    clientConfig:
      service:
        name: vault-agent-injector-svc
        namespace: default
        path: "/mutate"
      caBundle:
    rules:
      - operations: ["CREATE", "UPDATE"]
        apiGroups: [""]
        apiVersions: ["v1"]
        resources: ["pods"]

$ kubectl apply --filename mutatingwebhookconfigurations-vault-agent-injector.yaml

As per Vault documentation, “By default, the Vault Agent Injector will process all namespaces in Kubernetes except the system namespaces kube-system and kube-public. To limit what namespaces the injector can work in a namespace selector can be defined to match labels attached to namespaces.”

namespaceSelector is the selector for restricting the webhook to only specific namespaces. This should be set to a multiline string. For more information see https://kubernetes.io/docs/reference/access-authn-authz/extensible-admission-controllers/#matching-requests-namespaceselector

Deployment

$ cat deployment-vault-auto-injector.yaml
---
# Source: vault/templates/injector-deployment.yaml
# Deployment for the injector
apiVersion: apps/v1
kind: Deployment
metadata:
  name: vault-agent-injector
  namespace: default
  labels:
    app.kubernetes.io/name: vault-agent-injector
    app.kubernetes.io/instance: vault
    component: webhook
spec:
  replicas: 1
  selector:
    matchLabels:
      app.kubernetes.io/name: vault-agent-injector
      app.kubernetes.io/instance: vault
      component: webhook
  template:
    metadata:
      labels:
        app.kubernetes.io/name: vault-agent-injector
        app.kubernetes.io/instance: vault
        component: webhook
    spec:
      serviceAccountName: "vault-agent-injector"
      securityContext:
        runAsNonRoot: true
        runAsGroup: 1000
        runAsUser: 100
      containers:
        - name: sidecar-injector

          image: "hashicorp/vault-k8s:0.2.0"
          imagePullPolicy: "IfNotPresent"
          env:
            - name: AGENT_INJECT_LISTEN
              value: ":8080"
            - name: AGENT_INJECT_LOG_LEVEL
              value: info
            - name: AGENT_INJECT_VAULT_ADDR
              value: https://<VAULT_ADDR>:8200
            - name: AGENT_INJECT_VAULT_IMAGE
              value: "vault:1.3.2"
            - name: AGENT_INJECT_TLS_AUTO
              value: vault-agent-injector-cfg
            - name: AGENT_INJECT_TLS_AUTO_HOSTS
              value: vault-agent-injector-svc,vault-agent-injector-svc.default,vault-agent-injector-svc.default.svc
          args:
            - agent-inject
            - 2>&1
          livenessProbe:
            httpGet:
              path: /health/ready
              port: 8080
              scheme: HTTPS
            failureThreshold: 2
            initialDelaySeconds: 1
            periodSeconds: 2
            successThreshold: 1
            timeoutSeconds: 5
          readinessProbe:
            httpGet:
              path: /health/ready
              port: 8080
              scheme: HTTPS
            failureThreshold: 2
            initialDelaySeconds: 2
            periodSeconds: 2
            successThreshold: 1
            timeoutSeconds: 5

$ kubectl apply --filename deployment-vault-auto-injector.yaml

Note the value for AGENT_INJECT_VAULT_ADDR this will be unique for each Vault cluster. Also take notice of AGENT_INJECT_VAULT_IMAGE, this image must be present locally if restricting pods from only pulling images from a specific set of container repositories, such as GCR.

Kubernetes Auto-Injector Setup (Vault)

Enable and Configure Kubernetes Auth Method

$ export VAULT_SA_NAME=$(kubectl get sa vault-auth -o jsonpath="{.secrets[*]['name']}")
$ export SA_JWT_TOKEN=$(kubectl get secret $VAULT_SA_NAME -o jsonpath="{.data.token}" | base64 --decode; echo)
$ export SA_CA_CRT=$(kubectl get secret $VAULT_SA_NAME -o jsonpath="{.data['ca\.crt']}" | base64 --decode; echo) 

# determine Kubernetes master IP address (no https://) via `kubectl cluster-info`
$ export K8S_HOST=<K8S_MASTER_IP>

# set VAULT_TOKEN & VAULT_ADDR before next steps
$ vault auth enable kubernetes
$ vault write auth/kubernetes/config \
        token_reviewer_jwt="$SA_JWT_TOKEN" \
        kubernetes_host="https://$K8S_HOST:443" \
        kubernetes_ca_cert="$SA_CA_CRT"

Couple things with noting.

VAULT_TOKEN and VAULT_ADDR environment variables must be set before trying to communicate with the Vault server.

Be aware that the vault-auth service account is associated with the authentication method. This is the service account created explicitly for the Kubernetes authentication. It is not to be associated with any other resource or action. As such, should be protected accordingly.

If access to a Vault server is restricted by firewall rules and the Kubernetes cluster is not on the same internal network as the Vault server, it is critical a path is carved from the Kubernetes cluster to the Vault server, IE. whitelisting IP address of egress traffic from cluster in Vault’s firewall rule.

Example Application Steps

1. Enable KV (v2)

$ vault secrets enable -path=internal kv-v2

2. Store secret and password

$ vault kv put internal/database/config username="db-readonly-username" password="db-secret-password"

3. Create Vault policy

$ vault policy write internal-app - <<EOH
path "internal/data/database/config" {
  capabilities = ["read"]
}
EOH

4. Create Vault authentication role

$ vault write auth/kubernetes/role/internal-app \
        bound_service_account_names=internal-app \
        bound_service_account_namespaces=default \
        policies=internal-app \
        ttl=24h

5. Create Kubernetes service account

$ cat sa-internal-app.yml
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: internal-app
  
$ kubectl apply --filename sa-internal-app.yml

6. Create example application deployment (w/o annotations)

$ cat deployment-01-orgchart.yml
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: orgchart
  labels:
    app: vault-agent-injector-demo
spec:
  selector:
    matchLabels:
      app: vault-agent-injector-demo
  replicas: 1
  template:
    metadata:
      annotations:
      labels:
        app: vault-agent-injector-demo
    spec:
      serviceAccountName: internal-app
      containers:
        - name: orgchart
          image: jweissig/app:0.0.1
          
kubectl apply --filename deployment-01-orgchart.yml

At this point the pod does not have access to the secret, as there are no Vault specific annotations within the deployment, yet.

7. Patch deployment

$ cat deployment-02-inject-secrets.yml
---
spec:
  template:
    metadata:
      annotations:
        vault.hashicorp.com/agent-inject: "true"
        vault.hashicorp.com/role: "internal-app"
        vault.hashicorp.com/agent-inject-secret-database-config.txt: "internal/data/database/config"

$ kubectl patch deployment orgchart --patch "$(cat deployment-02-inject-secrets.yml)"

$ kubectl exec orgchart-<pod-id> --container orgchart -- cat /vault/secrets/database-config.txt
data: map[password:db-secret-password username:db-readonly-user]
metadata: map[created_time:2019-12-20T18:17:50.930264759Z deletion_time: destroyed:false version:2]

Now, there should be annotations associated with the pod. This will create the init and sidecar containers associated with the secret auto-injection. Secrets will be stored under /vault/secret. Worth noting, is the vault.hashicorp.com/agent-inject-secret-database-config.txt: "internal/data/database/config" annotation. The key dictates the name of the file to be database-config.txt with the value being the path within Vault’s KV store: internal/data/database/config.

8. Templatize secret output

$ cat deployment-03-inject-secrets-as-template.yml
---
spec:
  template:
    metadata:
      annotations:
        vault.hashicorp.com/agent-inject: "true"
        vault.hashicorp.com/agent-inject-status: "update"
        vault.hashicorp.com/role: "internal-app"
        vault.hashicorp.com/agent-inject-secret-database-config.txt: "internal/data/database/config"
        vault.hashicorp.com/agent-inject-template-database-config.txt: |
          {{- with secret "internal/data/database/config" -}}
          postgresql://{{ .Data.data.username }}:{{ .Data.data.password }}@postgres:5432/wizard
          {{- end -}}

$ kubectl patch deployment orgchart --patch "$(cat deployment-03-inject-secrets-as-template.yml)"

$ kubectl exec -it orgchart-<pod-id> -c orgchart -- cat /vault/secrets/database-config.txt
postgresql://db-readonly-user:db-secret-password@postgres:5432/wizard

There are multiple ways to configure how the secret is formatted, using the annotation template key vault.hashicorp.com/agent-inject-template-database-config.txt. For a complete list of annotation options, see the Vault Project links in the Attribution section below.

9. Create example application deployment (w/ annotations)

$ cat deployment-04-payrole.yml
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: payrole
  labels:
    app: vault-agent-injector-demo
spec:
  selector:
    matchLabels:
      app: vault-agent-injector-demo
  replicas: 1
  template:
    metadata:
      annotations:
        vault.hashicorp.com/agent-inject: "true"
        vault.hashicorp.com/agent-inject-status: "update"
        vault.hashicorp.com/role: "internal-app"
        vault.hashicorp.com/agent-inject-secret-database-config.txt: "internal/data/database/config"
        vault.hashicorp.com/agent-inject-template-database-config.txt: |
          {{- with secret "internal/data/database/config" -}}
          postgresql://{{ .Data.data.username }}:{{ .Data.data.password }}@postgres:5432/wizard
          {{- end -}}
      labels:
        app: vault-agent-injector-demo
    spec:
      serviceAccountName: internal-app
      containers:
        - name: payrole
          image: jweissig/app:0.0.1
          
$ kubectl apply --filename deployment-04-payrole.yml

Other Things Worth Noting

Secrets are bound to the service account. The service account for a deployment is specified in the spec.template.spec.serviceAccountName block. The corresponding Vault relation is specified in Vault’s authentication role

Secrets are bound to the namespace. Similar to the service account, the corresponding Vault relation is specified in Vault’s authentication role.

Errors for both service account and namespace will be manifest themselves as pods perpetually in the Init phase, never reaching ready. Errors are found within the vault-agent-init container logs, detailing the authentication error in clear English.

The token, both init and sidecar containers use to communicate with Vault, lives locally within the container at the following path: /home/vault/.token. Unsurprisingly, the token is not mounted into the primary container within the pod, making direct communications between Vault and primary container difficult.

[nguyenduybinh]

@ls-brentsmith, there's no annotation currently that lets you alter the auth path, however, you can mount your own Vault Agent configuration files using a configmap. Here's an example: https://www.vaultproject.io/docs/platform/k8s/injector/examples.html#configmap-example. Mount path would go in the method section: https://www.vaultproject.io/docs/agent/autoauth/index.html#mount_path

Hope that helps!

Hi,

I found an annotations option that help to custom the auth_path that easier than using config map.
You can use the annotation option of vault like:

vault.hashicorp.com/auth-path: "/auth/gcp-prod-k8s-cluster/"

Config annotation like that, the vault agent will call to endpoint: https://<vault_address>:8200/v1/auth/gcp-prod-k8s-cluster/login instead of default like that https://<vault_address>:8200/v1/auth/kubernetes/login

It's very useful when use have 1 vault cluster, and config authen with multiple k8s cluster.

Hope this help!

[burtlo]

Thanks @dcatalano-figure for the write up with the demonstrable code.

I recently delivered a HashiCorp Learn guide titled Integrate a Kubernetes Cluster with an External Vault.

In this guide, you will run Vault locally, start a Kubernetes cluster with Minikube, deploy an application that retrieves secrets from this Vault, and configure an injector only deployment to inject secrets into the pods from this Vault.

• Start Vault locally to act as the External Vault
• Start Minikube
• Determine the Vault address
• Deploy application with hard-coded Vault address
• Deploy service and endpoints to address an external Vault
• Define a Kubernetes service account
• Configure Kubernetes authentication
• Install the Vault Helm chart

[ls-brentsmith]

@ls-brentsmith, there's no annotation currently that lets you alter the auth path, however, you can mount your own Vault Agent configuration files using a configmap. Here's an example: https://www.vaultproject.io/docs/platform/k8s/injector/examples.html#configmap-example. Mount path would go in the method section: https://www.vaultproject.io/docs/agent/autoauth/index.html#mount_path
Hope that helps!

Hi,

I found an annotations option that help to custom the auth_path that easier than using config map.
You can use the annotation option of vault like:

vault.hashicorp.com/auth-path: "/auth/gcp-prod-k8s-cluster/"

Config annotation like that, the vault agent will call to endpoint: https://<vault_address>:8200/v1/auth/gcp-prod-k8s-cluster/login instead of default like that https://<vault_address>:8200/v1/auth/kubernetes/login

It's very useful when use have 1 vault cluster, and config authen with multiple k8s cluster.

Hope this help!

@ls-brentsmith, there's no annotation currently that lets you alter the auth path, however, you can mount your own Vault Agent configuration files using a configmap. Here's an example: https://www.vaultproject.io/docs/platform/k8s/injector/examples.html#configmap-example. Mount path would go in the method section: https://www.vaultproject.io/docs/agent/autoauth/index.html#mount_path
Hope that helps!

Hi,

I found an annotations option that help to custom the auth_path that easier than using config map.
You can use the annotation option of vault like:

vault.hashicorp.com/auth-path: "/auth/gcp-prod-k8s-cluster/"

Config annotation like that, the vault agent will call to endpoint: https://<vault_address>:8200/v1/auth/gcp-prod-k8s-cluster/login instead of default like that https://<vault_address>:8200/v1/auth/kubernetes/login

It's very useful when use have 1 vault cluster, and config authen with multiple k8s cluster.

Hope this help!

Indeed, this is a new feature in 0.2.0

[sandeepkatthi99]

I see this issue is resolved but could someone guide me to the final documentation. We have an external vault gke cluster and multiple application gke clusters. I would like to see how we can use this tool to integrate our existing vault to inject secrets to all those clusters.

[jasonodonnell]

@sandeepkatthi99: https://learn.hashicorp.com/vault/getting-started-k8s/external-vault

[sandeepkatthi99]

@jasonodonnell , as per this documentation it is using a minikube and it is using http protocol to connect to the external vault sever. Mine is an external vault server with https protocol which uses a ca authorized certificate. in this case how to change the helm command to create injector pod with certificate options? Please help me. If i just replace http with https I am gtting below error.

auth.handler: authenticating
2020-03-26T10:03:01.452Z [ERROR] auth.handler: error authenticating: error="Put https://****/v1/auth/kubernetes/login: x509: certificate signed by unknown authority" backoff=2.96206

I have built my vault cluster using https://codelabs.developers.google.com/codelabs/vault-on-gke/index.html?index=..%2F..cloud#18

[jasonodonnell]

Hi @sandeepkatthi99,

You need to create a Kube secret with the servers CA certificate (in the namespace with your app), then using annotations configure the Vault Agent to use it:

vault.hashicorp.com/tls-secret: "<NAME OF KUBE SECRET>"
vault.hashicorp.com/ca-cert: "/vault/tls/ca.crt"

[smurfralf]

This issue should be closed since the original description has been satisfied by vault-helm v0.4 with injector.externalVaultAddr value.

[sandeepkatthi99]

Thanaks @jasonodonnell jasonodonnell I will try the same.

[kosyfrances]

@burtlo Thanks for the learning guide.

I just wanted to let you know that in the Configure Kubernetes authentication section, the Kubernetes authentication link pointing to https://www.vaultproject.io/docs/auth/kubernetes.html displays a "Page Not Found".

[jasonodonnell]

There's a bug in the Vault website generator causing a bunch links to break and is something we're working on fixing. https://www.vaultproject.io/docs/auth/kubernetes/ will work and sorry for the inconvenience!

[therealsamlin]

@dcatalano-figure thanks for sharing your setup! I have a question though - I'm trying to go through with your setup however we have multiple clusters that would like to access vault secret. How do we go about that?

[stevegore]

@therealsamlin we're mounting the Kubernetes auth method at a different path for each cluster. Then set vault.hashicorp.com/auth-path appropriately on the injector per cluster.

[dcatalano-figure]

@dcatalano-figure thanks for sharing your setup! I have a question though - I'm trying to go through with your setup however we have multiple clusters that would like to access vault secret. How do we go about that?

basically what @stevegore said. would like to add that auth path value must be prefixed with auth/ in where if the mount was named k8s-cluster-01 the value would be auth/k8s-cluster-01

[ASangave]

In my case, I have vault server running on GKE cluster A and I have installed Vault Agent Injector service on cluster B by passing externalVaultAddr which is exposed over http.
But after deploying sample application on B along with necessary annotations to inject vault secrets, I am getting below error in vault-agent-init container of my sample app

2020-08-26T08:24:59.554Z [ERROR] auth.handler: error authenticating: error="Error making API request.

URL: PUT http://35.225.127.203/v1/auth/kubernetes/login
Code: 403. Errors:

* permission denied" backoff=2.908150444
2020-08-26T08:25:02.463Z [INFO]  auth.handler: authenticating

@jasonodonnell Any idea how to resolve this?

[asl-cloud99]

Hi, I am also facing the same issue. (I have 2 gke cluster setup , Cluster A vault is installed with internal LB, cluster B for application

Error writing data to auth/kubernetes/login: Error making API request.

URL: PUT http://gke_internal_LB_IP:8200/v1/auth/kubernetes/login
Code: 403. Errors:

• permission denied

and logs
[ERROR] auth.kubernetes.auth_kubernetes_c78cbc33: login unauthorized due to: Post "https://cluster_app_ip/apis/authentication.k8s.io/v1/tokenreviews": dial tcp cluster_app_ip:443: i/o timeout

[bakshigit]

Hello,

This question may have been answered earlier - not sure.

While performing the setup on EKS side , after executing all the 4 yaml files under deploy folder, i am able to create the agent injector POD.

However, when trying to perform the setup on the vault side, i am trying to obtain the JWT token as below , it complains of the missing secret token in secrets :

export SERVICEACCOUNT_JWT_TOKEN=$(kubectl get secret -o jsonpath="{.data.token}" $(kubectl get serviceaccount vault-injector -n vault -o jsonpath="{.secrets[0].name}") | base64 -d)
Error from server (NotFound): secrets "vault-injector-token-p5jjp" not found

I verify that the secret does in exist with kubectl get secrets -n vault command

Does the secret need to be created manually ? Is there a yaml to execute while deploying the agent injector itself ?

Please let me know.

[mgfnv9]

Hi @sandeepkatthi99,

You need to create a Kube secret with the servers CA certificate (in the namespace with your app), then using annotations configure the Vault Agent to use it:

vault.hashicorp.com/tls-secret: "<NAME OF KUBE SECRET>"
vault.hashicorp.com/ca-cert: "/vault/tls/ca.crt"

Hello, @jasonodonnell

please explain how to do it, I am new in Vault

[msenmurugan]

@asl-cloud99 We are also having the similar use case (I have 2 gke cluster setup with same VPC, Cluster A vault is installed with internal LB, cluster B for application). Are you able to make it to work? If so, can you please share the steps?

[jghal]

I deployed the injector via the standard helm chart, with --set injector.externalVaultAddr as describe in the Integrate a Kubernetes Cluster with an External Vault tutorial, and I'm seeing a TLS handshake error in the injector log whenever I deploy an annotated POD (the devwebapp-with-annotations from the tutorial).

[centos@ip-10-5-58-194 (vaulttest) ~]$ kubectl logs vault-agent-injector-5657ff569b-kdn8n -c sidecar-injector
2021-06-15T19:35:24.646Z [INFO]  handler: Starting handler..
Listening on ":8080"...
2021-06-15T19:35:24.654Z [INFO]  handler.auto-tls: Generated CA
2021-06-15T19:35:24.655Z [INFO]  handler.certwatcher: Updated certificate bundle received. Updating certs...
2021-06-16T17:11:24.000Z [INFO]  handler.certwatcher: Updated certificate bundle received. Updating certs...
2021-06-17T14:47:24.000Z [INFO]  handler.certwatcher: Updated certificate bundle received. Updating certs...
2021-06-18T12:23:24.000Z [INFO]  handler.certwatcher: Updated certificate bundle received. Updating certs...
2021-06-18T15:11:36.470Z [ERROR] handler: http: TLS handshake error from 127.0.0.1:56344: EOF

I've plenty of other TLS handshake issues reported with a bad certificate error but not this EOF variant.

[jghal]

A colleague "changed service mesh mtls strict policy to permissive" and that got rid of the TLS handshake error but the injector still wasn't injecting the init/agent containers. Then she disabled the istio sidecar from the vault-agent-injector POD, and it started working.

[MushiTheMoshi]

Hi Guys, I've tried all things shared here and still getting:

2021-06-20T21:38:24.892Z [INFO]  auth.handler: authenticating
2021-06-20T21:39:24.893Z [ERROR] auth.handler: error authenticating: error="context deadline exceeded" backoff=1.87s

HELM:
helm install vault hashicorp/vault --set
"server.dev.enabled=true,server.extraEnvironmentVars.VAULT_DEV_LISTEN_ADDRESS=0.0.0.0:8200,server.extraEnvironmentVars.VAULT_ADDR=http://vault:8200"

APP.YML:

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: vault-app
spec:
  selector:
    matchLabels:
      app: vault-app
  template:
    metadata:
      labels:
        app: vault-app
      annotations:
        vault.hashicorp.com/agent-inject: "true"
        vault.hashicorp.com/agent-init-first: "true"
        vault.hashicorp.com/agent-inject-secret-hello.txt: secret/hello
        vault.hashicorp.com/role: vault-app
        vault.hashicorp.com/agent-pre-populate: "false"
        vault.hashicorp.com/service: "http://vault:8200"
    spec:
      serviceAccountName: vault-app
      containers:
      - name: debian
        image: debian:latest
        command: [sleep, infinity]

---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: vault-app
  namespace: default
  labels: 
    app: vault-app

---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: vault-app
  namespace: default
  labels:
    app: vault-app

roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole 
  name: system:auth-delegator

subjects:
- kind: ServiceAccount
  namespace: default
  name: vault-app

---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-all-ingress
spec:
  podSelector: {}
  ingress:
  - {}
  policyTypes:
  - Ingress

VAULT_0 conf:

vault kv put secret/hello foo=world

vault auth enable kubernetes
vault write auth/kubernetes/config \
   token_reviewer_jwt="$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)" \
   kubernetes_host="https://$KUBERNETES_PORT_443_TCP_ADDR:443" \
   kubernetes_ca_cert=@/var/run/secrets/kubernetes.io/serviceaccount/ca.crt 
   # disable_local_ca_jwt=true \
   # disable_iss_validation=true 


vault policy write vault-app - <<'EOF'
path "secret/hello" {  
   capabilities = ["create", "read", "update", "delete", "list"]
}
EOF

vault write auth/kubernetes/role/vault-app \
   bound_service_account_names=vault-app \
   bound_service_account_namespaces=* \
   policies=vault-app ttl=24h

Would you please help me to spot what is wrong here? I'd really really appreciate some help ...
P.D: I've check all the rolebindings + services accounts + webmutations elated to vault and all of them seem to be correct ...

[raghuramopsmx]

Hi,
I followed https://learn.hashicorp.com/tutorials/vault/kubernetes-external-vault?in=vault/kubernetes#install-the-vault-helm-chart-configured-to-address-an-external-vault this document.
Installed Vault on one cluster and trying to retrieve secrets into the pod which is deployed on another cluster.

However, I am facing authentication issue as shown below

2021-12-22T11:55:02.769Z [ERROR] auth.handler: error authenticating:
error=
| Error making API request.
|
| URL: PUT http://51.143.61.191:8200/v1/auth/kubernetes/login
| Code: 403. Errors:
|
| * permission denied
backoff=4m1.09s

I have enabled debug, here is the output of it

2021-12-22T11:08:10.887Z [DEBUG] (runner) final config: {"Consul":{"Address":"","Namespace":"","Auth":{"Enabled":false,"Username":"","Password":""},"Retry":{"Attempts":12,"Backoff":250000000,"MaxBackoff":60000000000,"Enabled":true},"SSL":{"CaCert":"","CaPath":"","Cert":"","Enabled":false,"Key":"","ServerName":"","Verify":true},"Token":"","Transport":{"CustomDialer":null,"DialKeepAlive":30000000000,"DialTimeout":30000000000,"DisableKeepAlives":false,"IdleConnTimeout":90000000000,"MaxIdleConns":100,"MaxIdleConnsPerHost":9,"TLSHandshakeTimeout":10000000000}},"Dedup":{"Enabled":false,"MaxStale":2000000000,"Prefix":"consul-template/dedup/","TTL":15000000000,"BlockQueryWaitTime":60000000000},"DefaultDelims":{"Left":null,"Right":null},"Exec":{"Command":"","Enabled":false,"Env":{"Denylist":[],"Custom":[],"Pristine":false,"Allowlist":[]},"KillSignal":2,"KillTimeout":30000000000,"ReloadSignal":null,"Splay":0,"Timeout":0},"KillSignal":2,"LogLevel":"DEBUG","MaxStale":2000000000,"PidFile":"","ReloadSignal":1,"Syslog":{"Enabled":false,"Facility":"LOCAL0","Name":"consul-template"},"Templates":[{"Backup":false,"Command":"","CommandTimeout":30000000000,"Contents":"{{ with secret "secret/data/devwebapp/config" }}{{ range $k, $v := .Data }}{{ $k }}: {{ $v }}\n{{ end }}{{ end }}","CreateDestDirs":true,"Destination":"/vault/secrets/credentials.txt","ErrMissingKey":false,"Exec":{"Command":"","Enabled":false,"Env":{"Denylist":[],"Custom":[],"Pristine":false,"Allowlist":[]},"KillSignal":2,"KillTimeout":30000000000,"ReloadSignal":null,"Splay":0,"Timeout":30000000000},"Perms":0,"Source":"","Wait":{"Enabled":false,"Min":0,"Max":0},"LeftDelim":"{{","RightDelim":"}}","FunctionDenylist":[],"SandboxPath":""}],"Vault":{"Address":"http://51.143.61.191:8200","Enabled":true,"Namespace":"","RenewToken":false,"Retry":{"Attempts":12,"Backoff":250000000,"MaxBackoff":60000000000,"Enabled":true},"SSL":{"CaCert":"","CaPath":"","Cert":"","Enabled":false,"Key":"","ServerName":"","Verify":false},"Transport":{"CustomDialer":null,"DialKeepAlive":30000000000,"DialTimeout":30000000000,"DisableKeepAlives":false,"IdleConnTimeout":90000000000,"MaxIdleConns":100,"MaxIdleConnsPerHost":9,"TLSHandshakeTimeout":10000000000},"UnwrapToken":false,"DefaultLeaseDuration":300000000000},"Wait":{"Enabled":false,"Min":0,"Max":0},"Once":false,"BlockQueryWaitTime":60000000000}

Could someone please help in resolving this issue...

[bakshigit]

Hi, I followed https://learn.hashicorp.com/tutorials/vault/kubernetes-external-vault?in=vault/kubernetes#install-the-vault-helm-chart-configured-to-address-an-external-vault this document. Installed Vault on one cluster and trying to retrieve secrets into the pod which is deployed on another cluster.

However, I am facing authentication issue as shown below

2021-12-22T11:55:02.769Z [ERROR] auth.handler: error authenticating: error= | Error making API request. | | URL: PUT http://51.143.61.191:8200/v1/auth/kubernetes/login | Code: 403. Errors: | | * permission denied backoff=4m1.09s

I have enabled debug, here is the output of it

2021-12-22T11:08:10.887Z [DEBUG] (runner) final config: {"Consul":{"Address":"","Namespace":"","Auth":{"Enabled":false,"Username":"","Password":""},"Retry":{"Attempts":12,"Backoff":250000000,"MaxBackoff":60000000000,"Enabled":true},"SSL":{"CaCert":"","CaPath":"","Cert":"","Enabled":false,"Key":"","ServerName":"","Verify":true},"Token":"","Transport":{"CustomDialer":null,"DialKeepAlive":30000000000,"DialTimeout":30000000000,"DisableKeepAlives":false,"IdleConnTimeout":90000000000,"MaxIdleConns":100,"MaxIdleConnsPerHost":9,"TLSHandshakeTimeout":10000000000}},"Dedup":{"Enabled":false,"MaxStale":2000000000,"Prefix":"consul-template/dedup/","TTL":15000000000,"BlockQueryWaitTime":60000000000},"DefaultDelims":{"Left":null,"Right":null},"Exec":{"Command":"","Enabled":false,"Env":{"Denylist":[],"Custom":[],"Pristine":false,"Allowlist":[]},"KillSignal":2,"KillTimeout":30000000000,"ReloadSignal":null,"Splay":0,"Timeout":0},"KillSignal":2,"LogLevel":"DEBUG","MaxStale":2000000000,"PidFile":"","ReloadSignal":1,"Syslog":{"Enabled":false,"Facility":"LOCAL0","Name":"consul-template"},"Templates":[{"Backup":false,"Command":"","CommandTimeout":30000000000,"Contents":"{{ with secret "secret/data/devwebapp/config" }}{{ range $k, $v := .Data }}{{ $k }}: {{ $v }}\n{{ end }}{{ end }}","CreateDestDirs":true,"Destination":"/vault/secrets/credentials.txt","ErrMissingKey":false,"Exec":{"Command":"","Enabled":false,"Env":{"Denylist":[],"Custom":[],"Pristine":false,"Allowlist":[]},"KillSignal":2,"KillTimeout":30000000000,"ReloadSignal":null,"Splay":0,"Timeout":30000000000},"Perms":0,"Source":"","Wait":{"Enabled":false,"Min":0,"Max":0},"LeftDelim":"{{","RightDelim":"}}","FunctionDenylist":[],"SandboxPath":""}],"Vault":{"Address":"http://51.143.61.191:8200","Enabled":true,"Namespace":"","RenewToken":false,"Retry":{"Attempts":12,"Backoff":250000000,"MaxBackoff":60000000000,"Enabled":true},"SSL":{"CaCert":"","CaPath":"","Cert":"","Enabled":false,"Key":"","ServerName":"","Verify":false},"Transport":{"CustomDialer":null,"DialKeepAlive":30000000000,"DialTimeout":30000000000,"DisableKeepAlives":false,"IdleConnTimeout":90000000000,"MaxIdleConns":100,"MaxIdleConnsPerHost":9,"TLSHandshakeTimeout":10000000000},"UnwrapToken":false,"DefaultLeaseDuration":300000000000},"Wait":{"Enabled":false,"Min":0,"Max":0},"Once":false,"BlockQueryWaitTime":60000000000}

Could someone please help in resolving this issue...

Suggest you check the auth Method of the Cluster from where you are trying to retrieve secret.