Client cert returned by exec not used in TLS connection

[rcanderson23]

Current and expected behavior

Current behavior:

• When using an exec command that returns a client cert and key, client auth is not used.

2022-11-19T21:01:49.398808Z DEBUG rustls::client::common: Client auth requested but no cert/sigscheme available

Expected behavior:

• When using an exec command that returns a client cert and key, cert is used for authn.

Possible solution

I was able to implement a fix locally rcanderson23@087f5bf. This is going through Teleport using tsh kube credentials to return the client cert and key. With the above configured and a custom client with tls server name overridden it works. This fix seems less than ideal though as I believe this running exec twice, first for the tls credentials and second for any tokens that could get passed in as AuthLayers.

Technically, these certs are short lived (8-12 hours typically) and the ExecStatus returns an expiration timestamp. As I understand it, supporting this also would require having to create a new client if I understand this correctly. I'm not sure that is worth attempting to tackle though.

Additional context

No response

Environment

➜ kubectl version --short
Client Version: v1.25.4
Kustomize Version: v4.5.7
Server Version: v1.25.3

Configuration and features

No response

Affected crates

kube-client

Would you like to work on fixing this bug?

yes

[clux]

Hey, thanks for digging into this!

This was confusing me a bit. The way I expected this to be fixed was to extend RefreshableToken::Exec

kube/kube-client/src/client/auth/mod.rs

Line 152 in be07824

Exec(Arc<Mutex<(SecretString, DateTime<Utc>, AuthInfo)>>),

to be able to pass back certs so that they could be hooked up through the Layer. But maybe that is too late since the underlying Connector is already made and configured at that point?

I think it's worth opening up a PR with what you got so we can comment on it a little more specifically (for one, it currently seems to default to try exec first, which probably is wrong?). Am curious how the user should hook this up though, since you've made a new user auth type for this and exposed a ConfigExt method. Is the idea that users get 8-12 hours of connectivity and then they have to restart, or does this work indefinitely upstream via kubectl? Depending on how the layer is configured, you might get 1 exec call every request, or just one at client startup. The normal refreshable exec setup deals with timeouts for this.

Note that we do already have an extra client being created in the oauth module (oauth.rs) for exec anyway.

[kazk]

I don't think we can support this cleanly at the moment. Layers we use are middlewares for requests, and this requires changing the client.

I think the hyper client instance in kube::Client must be replaced whenever the client certificate and/or key changes. This is different from oauth which creates a new hyper client to get a token, then uses that to make requests with the main client.

If we don't need to worry about an expiring client certificate (e.g., this is used for development only and certificates lasts long enough), we can provide a workaround to call the exec plugin once that updates the config's client certificate/key before creating the client.

[rcanderson23]

I don't think we can support this cleanly at the moment. Layers we use are middlewares for requests, and this requires changing the client.

I think the hyper client instance in kube::Client must be replaced whenever the client certificate and/or key changes. This is different from oauth which creates a new hyper client to get a token, then uses that to make requests with the main client.

If we don't need to worry about an expiring client certificate (e.g., this is used for development only and certificates lasts long enough), we can provide a workaround to call the exec plugin once that updates the config's client certificate/key before creating the client.

This was my take away (however I am new to Rust so I was ready to be wrong). I think a workaround calling out this limitation is fine enough imo. I think the primary use case for this type of exec is CLI application which will refresh the cert if needed and with local dev it is easy enough to refresh your cert as needed.