backend: oidc: impersonate instead of forwarding oidc token to k8s api #2814

iSE-stefan-may · 2025-01-30T16:38:55Z

By default, the OIDC JWT is forwarded to the Kubernetes API server. In some cases it might not be possible to have the Kubernetes API server configured with OIDC, in these cases the OIDC JWT should be evaluated by Headlamp and the request should use the service account of the pod together with impersonating the user in the token.

Related issue:

Support OIDC without the Kuberentes API Server configuration (something similar to Argo CD configuration) #2614

Testing

To test this PR, do the following:

build the container image
follow the documentation at https://headlamp.dev/docs/latest/installation/in-cluster/keycloak/ for setting up a keycloak client and configuring a clusterrole binding (do NOT follow the steps for setting up minikube)
deploy the helm chart with the oidc values and oidc.impersonate=true and oidc.impersonateClaim=preferred_username using the container image from step 1
go to headlamp and login with oidc

zivcex

lg2m

illume

Hello. Thanks for this!

I left a few questions/notes.

Can you please add some steps in the PR description on how we can test this?

Would it be possible for you to add some test cases?

illume · 2025-02-05T10:31:19Z

backend/cmd/headlamp.go

+		}
+
+		idToken, err := c.getValidToken(r.Context(), token)
+		if err != nil {


Can you please add some logging in the error conditions?

Logging added, but I'm not really sure, whether it is really the best case to log always when the token validation fails. It may also be, that OIDC impersonation is active, but the user still used a serviceaccount token, in those cases a warning in the log would be irritating.

Ok. That sounds like it makes sense to remove.

Is something like this possible? Otherwise remove that logging?

if ! OIDCImpersonationEnaled and serviceAccount { log() }

Wether the user provided token is a kubernetes serviceaccount token can only be evaluated from the Kubernetes API, therefore this check won't be possible. Only option where this would be possible would be, if we disable the option to login in the ui via a serviceaccount token altogether. In this case we could give a log warning here, but as far as I can see, that's currently not possible.
I'll cleanup the logging for the cases, where it may not be an error case.

illume · 2025-02-05T10:33:25Z

backend/cmd/headlamp.go

 func StartHeadlampServer(config *HeadlampConfig) {
 	handler := createHeadlampHandler(config)

 	handler = config.OIDCTokenRefreshMiddleware(handler)
+	if config.oidcImpersonate {
+		handler = config.OIDCImpersonateMiddleware(handler)


Can you please explain why this middleware is after the OIDCTokenRefreshMiddleware?

It was the most fitting position I thought of, but not a particular reason. Do you have better recommendations?

I was wondering how it would interact with the other middleware? Wondering if it would make sense to go before OIDCTokenRefreshMiddleware or not?

Because would the token refresh need to use the impersonated stuff? If that's the case I was wondering if the OIDCImpersonateMiddleware should go first?

You are right, the OIDCTokenRefreshMiddleware needs the original token, therefore it must be evaluated before OIDCImpersonateMiddleware. I'll change the order of those two and add a comment.

illume · 2025-02-05T10:35:19Z

backend/pkg/config/config.go

@@ -166,6 +168,10 @@ func flagset() *flag.FlagSet {
 	f.String("oidc-idp-issuer-url", "", "Identity provider issuer URL for OIDC")
 	f.String("oidc-scopes", "profile,email",
 		"A comma separated list of scopes needed from the OIDC provider")
+	f.Bool("oidc-impersonate", false,


@knrt10 would this need to be added to the helm chart as well?

Added to the helm chart, however I'm not convinced that I did it in a good way, since the chart is a little bit confusing for me. So please recheck.

Our main helm chart expert is on holidays at the moment. So this will take a bit longer to review. Bear with me :)

backend/cmd/headlamp.go

illume · 2025-02-05T11:26:42Z

Some lint issues in CI job: https://github.com/headlamp-k8s/headlamp/actions/runs/13056969725/job/36711540140?pr=2814#step:7:27

Signed-off-by: Stefan May <stefan.may@ise-online.com>

iSE-stefan-may · 2025-02-06T16:44:32Z

Have updated some code regarding the comments, but not fully tested yet and not added a test description. Will work on that tomorrow, but perhaps you can already review the answers to the comments.

audunsolemdal · 2025-02-06T18:17:55Z

Hi, not an expert here. I am wondering if this implementation will support impersonation based on groups listed in the groups claim found in the token returned from the OIDC provider?

illume · 2025-02-07T07:28:23Z

Have updated some code regarding the comments, but not fully tested yet and not added a test description. Will work on that tomorrow, but perhaps you can already review the answers to the comments.

Thanks. Reading through it now :)

Signed-off-by: Stefan May <stefan.may@ise-online.com>

zivcex · 2025-02-07T11:08:14Z

charts/headlamp/templates/deployment.yaml

@@ -147,11 +177,21 @@ spec:
            # Check if scopes are non empty either from env or oidc.config
            - "-oidc-scopes=$(OIDC_SCOPES)"
            {{- end }}
+            {{- if or ($oidc.impersonate) ($impersonate) }}


Hey great PR!!

I'm wondering would it be too heavy if you can provide the groups as well here within the chart and deployment?

more details are here:
https://kubernetes.io/docs/reference/access-authn-authz/authentication/#user-impersonation:~:text=Impersonate%2DGroup%3A%20A%20group%20name%20to%20act%20as.%20Can%20be%20provided%20multiple%20times%20to%20set%20multiple%20groups.%20Optional.%20Requires%20%22Impersonate%2DUser%22.

Convention here is that groups claim in JWT hold the user group coming in from IDP. It would be good if it was configurable as some IDPs send the groups membership under other claim name.

I have added functionality to use groups too and changed the parameters therefore to -oidc-user-claim and -oidc-groups-claim. Also added a filter for additional impersonate-* headers to prevent any authentication attacks, since the service account token is now used.

illume · 2025-02-08T04:37:50Z

It looks like the backend/lint CI job is failing.

You can run make backend-lint locally.

illume · 2025-02-08T04:38:19Z

The helm template job is having an issue: https://github.com/headlamp-k8s/headlamp/actions/runs/13197046279/job/36885634312?pr=2814

Template test failed for ./charts/headlamp/tests/test_cases/oidc-create-secret.yaml against ./charts/headlamp/tests/expected_templates/oidc-create-secret.yaml:
--- ./charts/headlamp/tests/oidc-create-secret.yaml_output/rendered_templates.yaml	2025-02-08 02:48:20.528327104 +0000
+++ ./charts/headlamp/tests/expected_templates/oidc-create-secret.yaml	2025-02-08 02:48:20.536327[13](https://github.com/headlamp-k8s/headlamp/actions/runs/13197046279/job/36885634312?pr=2814#step:5:14)6 +0000
@@ -126,11 +126,6 @@
                 secretKeyRef:
                   name: oidc
                   key: scopes
-            - name: OIDC_IMPERSONATE_CLAIM
-              valueFrom:
-                secretKeyRef:
-                  name: oidc
-                  key: impersonateClaim
make: *** [Makefile:[18](https://github.com/headlamp-k8s/headlamp/actions/runs/13197046279/job/36885634312?pr=2814#step:5:19)5: helm-template-test] Error 1
           args:
             - "-in-cluster"
             - "-plugins-dir=/headlamp/plugins"
Error: Process completed with exit code 2.

To test the charts locally

make helm-template-test

Signed-off-by: Stefan May <stefan.may@ise-online.com>

mlbiam · 2025-02-11T00:55:22Z

By default, the OIDC JWT is forwarded to the Kubernetes API server. In some cases it might not be possible to have the Kubernetes API server configured with OIDC, in these cases the OIDC JWT should be evaluated by Headlamp and the request should use the service account of the pod together with impersonating the user in the token.

This is not a good idea. Dashboards should never be configured with privileged token. it's too easy to expose a dashbaord by accident. Even when impersonating the user, headlamp should rely on the token from the proxy, not its own. This way someone accidentally exposes the dashbaord outside of the proxy, they haven't created a vector to access the API server with a privleged account.

illume · 2025-02-11T12:18:24Z

This is not a good idea. Dashboards should never be configured with privileged token. it's too easy to expose a dashbaord by accident. Even when impersonating the user, headlamp should rely on the token from the proxy, not its own. This way someone accidentally exposes the dashbaord outside of the proxy, they haven't created a vector to access the API server with a privleged account.

It is a good idea to try and help people prevent making mistakes with configuration accidentally opening things up.

Also, when hearing about other peoples use cases it is often the situation that they are best placed to evaluate and decide why they want to do things in a particular way.

@iSE-stefan-may maybe you could give us some ideas about this? Could you elaborate on your use case a bit? Also, what do you think about this possibility of opening things up accidentally and how we could address this?

joaquimrocha · 2025-02-14T10:50:08Z

By default, the OIDC JWT is forwarded to the Kubernetes API server. In some cases it might not be possible to have the Kubernetes API server configured with OIDC, in these cases the OIDC JWT should be evaluated by Headlamp and the request should use the service account of the pod together with impersonating the user in the token.

This is not a good idea. Dashboards should never be configured with privileged token. it's too easy to expose a dashbaord by accident. Even when impersonating the user, headlamp should rely on the token from the proxy, not its own. This way someone accidentally exposes the dashbaord outside of the proxy, they haven't created a vector to access the API server with a privleged account.

So far we have avoided requiring or encouraging running Headlamp with its own token/role. We did have some users asking for this and we resorted to advising to use a reverse proxy. We even have thought of maybe having a side project to facilitate this for Headlamp; would this bring any advantage here?

@mlbiam I'd like to ask your opinion whether this is still a security concern if we add the feature as optional and somehow make it clear, via CLI options' names, extra options, and documentation what the implications of this are. What would that look like?

mlbiam · 2025-02-14T12:25:47Z

We did have some users asking for this and we resorted to advising to use a reverse proxy.

Just because users want it doesn't mean its a good idea. Users in general tend to rely on what they can easily control, and a port-forward to a dashboard that is privileged is something they control. Unfortunately, it's also something anyone else can control too! This was how Tesla was breached (https://www.trendmicro.com/vinfo/us/security/news/cybercrime-and-digital-threats/tesla-and-jenkins-servers-fall-victim-to-cryptominers) and has led to the assumption by security teams that the Kubernetes dashboard is "insecure". It's also the reason why the Kubernetes Dashboard moved away from being able to use its own ServiceAccount token and requiring a reverse proxy to inject a token in the 7.x version.

Headlamp already has a "local" mode. If users don't want to do it "correctly" in a centralized way, they should use the local version.

We even have thought of maybe having a side project to facilitate this for Headlamp; would this bring any advantage here?

easier with good security is always better. That said, i don't know you need a new project. OpenUnison does this today. You can also use oauth2proxy+kube-oidc-proxy too if you don't want to use openunison: https://www.tremolo.io/post/kubernetes-authentication-comparing-solutions

That said, you're essentially talking about rolling your own authentication code, which i'd recommend against. Best to invest time in good docs, better helm charts, etc, for existing projects to make this easier. The hard part isn't the proxies, it's setting up DNS, TLS, integration with a authentication team that is in a different silo in your org, etc. Creating another project won't fix those issues, just create one more thing to support.

I'd like to ask your opinion whether this is still a security concern if we add the feature as optional and somehow make it clear, via CLI options' names, extra options, and documentation what the implications of this are

I'd still say this is an anti-pattern. All the red flags and warnings won't stop users from doing the easy thing. ie: clicking through a security warning for an invalid certificate. It will only take one major breach for Headlamp to earn an "insecure" badge.

The Kubernetes Dashbaord moved awat from this pattern for 7.x and has received considerable pushback. Their answer (which I think is correct), is setup a reverse proxy that hard codes an SA in it. It's terribly insecure, but so's running the dashboard with a privileged sa. It at least makes it harder to do the insecure thing and protects the project's reputation.

zivcex · 2025-02-19T10:57:20Z

Hi @iSE-stefan-may will you close this one given the new circumstances? Or leave this setting optional?

aslafy-z · 2025-02-24T04:34:25Z

backend/cmd/headlamp.go

+		}
+		r.Header.Set("Impersonate-User", user)
+		for _, group := range groups {
+			r.Header.Set("Impersonate-Group", group)


I believe this should use Add instead of Set.

Suggested change

r.Header.Set("Impersonate-Group", group)

r.Header.Add("Impersonate-Group", group)

dosubot bot added the size:L This PR changes 100-499 lines, ignoring generated files. label Jan 30, 2025

iSE-stefan-may force-pushed the feat/oidc-impersonate branch from 8661611 to cdb40c2 Compare January 30, 2025 16:41

iSE-stefan-may mentioned this pull request Jan 31, 2025

Support OIDC without the Kuberentes API Server configuration (something similar to Argo CD configuration) #2614

Open

zivcex approved these changes Feb 4, 2025

View reviewed changes

illume requested changes Feb 5, 2025

View reviewed changes

umarhussain15 mentioned this pull request Feb 5, 2025

Secure access to in-cluster headlamp with a reverse proxy #2207

Open

backend: oidc: impersonate instead of forwarding oidc token to k8s api

a1bbd62

Signed-off-by: Stefan May <stefan.may@ise-online.com>

iSE-stefan-may force-pushed the feat/oidc-impersonate branch from cdb40c2 to a1bbd62 Compare February 6, 2025 16:36

iSE-stefan-may added 2 commits February 7, 2025 10:02

Merge branch 'main' into feat/oidc-impersonate

bd747b2

backend: oidc: adaptions

53fadbe

Signed-off-by: Stefan May <stefan.may@ise-online.com>

zivcex reviewed Feb 7, 2025

View reviewed changes

backend: oidc: cleanup, oidc groups added, filter for other headers

039dde1

Signed-off-by: Stefan May <stefan.may@ise-online.com>

aslafy-z reviewed Feb 24, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

backend: oidc: impersonate instead of forwarding oidc token to k8s api #2814

backend: oidc: impersonate instead of forwarding oidc token to k8s api #2814

iSE-stefan-may commented Jan 30, 2025 •

edited by illume

Loading

zivcex left a comment

illume left a comment

illume Feb 5, 2025

iSE-stefan-may Feb 6, 2025

illume Feb 8, 2025

iSE-stefan-may Feb 10, 2025

illume Feb 5, 2025

iSE-stefan-may Feb 6, 2025

illume Feb 8, 2025

iSE-stefan-may Feb 10, 2025

illume Feb 5, 2025

iSE-stefan-may Feb 6, 2025

illume Feb 8, 2025

illume commented Feb 5, 2025

iSE-stefan-may commented Feb 6, 2025

audunsolemdal commented Feb 6, 2025

illume commented Feb 7, 2025

zivcex Feb 7, 2025

iSE-stefan-may Feb 10, 2025

illume commented Feb 8, 2025 •

edited

Loading

illume commented Feb 8, 2025 •

edited

Loading

mlbiam commented Feb 11, 2025

illume commented Feb 11, 2025

joaquimrocha commented Feb 14, 2025

mlbiam commented Feb 14, 2025

zivcex commented Feb 19, 2025

aslafy-z Feb 24, 2025

	r.Header.Set("Impersonate-Group", group)
	r.Header.Add("Impersonate-Group", group)

backend: oidc: impersonate instead of forwarding oidc token to k8s api #2814

Are you sure you want to change the base?

backend: oidc: impersonate instead of forwarding oidc token to k8s api #2814

Conversation

iSE-stefan-may commented Jan 30, 2025 • edited by illume Loading

Testing

zivcex left a comment

Choose a reason for hiding this comment

illume left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

illume commented Feb 5, 2025

iSE-stefan-may commented Feb 6, 2025

audunsolemdal commented Feb 6, 2025

illume commented Feb 7, 2025

Choose a reason for hiding this comment

Choose a reason for hiding this comment

illume commented Feb 8, 2025 • edited Loading

illume commented Feb 8, 2025 • edited Loading

To test the charts locally

mlbiam commented Feb 11, 2025

illume commented Feb 11, 2025

joaquimrocha commented Feb 14, 2025

mlbiam commented Feb 14, 2025

zivcex commented Feb 19, 2025

Choose a reason for hiding this comment

iSE-stefan-may commented Jan 30, 2025 •

edited by illume

Loading

illume commented Feb 8, 2025 •

edited

Loading

illume commented Feb 8, 2025 •

edited

Loading