Help: can't write to secrets path and can't disable secrets path

[voiprodrigo]

Hi @sethvargo ,

I'm trying to setup your plugin with Vault 1.6.1.
I started by downloading the latest release file for Linux amd64, installing it in a Vault 1.6.0 cluster.
I registered the plugin and enabled a secrets path with it. Then tried to generate a password but it fails with 500/internal error.

At this point I proceeded with upgrading Vault to latest 1.6.1, compiling the plugin and re-deploy it. I de-registered and re-registered. Then tried again to generate password but fails in the same way.

I then tried to disable the secrets path using the plugin, but that fails with 400/unrecognized remote plugin message.

Vault logs show:

Dec 19 02:31:54 vaulthost vault[7577]: 2020-12-19T02:31:54.879Z [DEBUG] secrets.secrets-gen.secrets-gen_c4e68857.secrets-gen: starting plugin: path=/etc/vault/plugins/vault-secrets-gen args=[/etc/vault/plugins/vault-secrets-gen]
Dec 19 02:31:54 vaulthost vault[7577]: 2020-12-19T02:31:54.880Z [DEBUG] secrets.secrets-gen.secrets-gen_c4e68857.secrets-gen: plugin started: path=/etc/vault/plugins/vault-secrets-gen pid=9442
Dec 19 02:31:54 vaulthost vault[7577]: 2020-12-19T02:31:54.880Z [DEBUG] secrets.secrets-gen.secrets-gen_c4e68857.secrets-gen: waiting for RPC address: path=/etc/vault/plugins/vault-secrets-gen
Dec 19 02:31:55 vaulthost vault[7577]: 2020-12-19T02:31:55.001Z [INFO]  http: TLS handshake error from 1.2.3.4:32818: tls: client didn't provide a certificate
Dec 19 02:31:56 vaulthost vault[7577]: 2020-12-19T02:31:56.531Z [INFO]  http: TLS handshake error from 1.2.3.4:32820: tls: client didn't provide a certificate
Dec 19 02:31:56 vaulthost vault[7577]: 2020-12-19T02:31:56.949Z [INFO]  expiration: revoked lease: lease_id=sys/wrapping/wrap/h11181ea0a80c45d43bfd2f83be939622804527d376e6496f54969d8cfba73edc
Dec 19 02:31:58 vaulthost vault[7577]: 2020-12-19T02:31:58.992Z [INFO]  http: TLS handshake error from 1.2.3.4:32822: tls: client didn't provide a certificate
Dec 19 02:31:58 vaulthost vault[7577]: 2020-12-19T02:31:58.993Z [ERROR] secrets.secrets-gen.secrets-gen_c4e68857.secrets-gen.vault-secrets-gen: plugin tls init: error="error during token unwrap request: Put "https://a.b.c.d:8200/v1/sys/wrapping/unwrap": remote error: tls: bad certificate" timestamp=2020-12-19T02:31:58.992Z
Dec 19 02:31:58 vaulthost vault[7577]: 2020-12-19T02:31:58.995Z [ERROR] core: failed to run existence check: error="Unrecognized remote plugin message:
Dec 19 02:31:58 vaulthost vault[7577]: This usually means that the plugin is either invalid or simply
Dec 19 02:31:58 vaulthost vault[7577]: needs to be recompiled to support the latest protocol."
Dec 19 02:31:58 vaulthost vault[7577]: 2020-12-19T02:31:58.995Z [DEBUG] secrets.secrets-gen.secrets-gen_c4e68857.secrets-gen: plugin process exited: path=/etc/vault/plugins/vault-secrets-gen pid=9442

Any suggestion on what I might be doing wrong?
Thanks!

[voiprodrigo]

Also interesting:

$ vault plugin list | grep secrets-gen
secrets-gen
$ vault plugin info secrets-gen
No value found for plugin "secrets-gen"

[voiprodrigo]

Regarding the TLS errors, I am using TLS in the cluster indeed, and also the client is authenticating with cert method. But that has been working fine for everything else. api_addr is also set (don't know if relevant, but saw you mentioning it in another issue).

[voiprodrigo]

I was missing some stuff, of course.

The plugin commands needs to include the TLS related options, as I'm using TLS in the API listener. So I added -ca-cert, -client-cert and -client-key to the command. In fact ./plugin -h shows the options. So that addressed the TLS handshake errors I posted before. I would suggest this should be in the README.

Also, I wasn't invoking plugin info correctly, need to pass the plugin type before the plugin name, so the correct command is "valut plugin info secret secrets-gen/".

However, not quite there yet. If I try to disable the path, it still fails:

$ vault secrets list

Path            Type           Accessor                Description
----            ----           --------                -----------
cubbyhole/      cubbyhole      cubbyhole_74d9f642      per-token private secret storage
identity/       identity       identity_d4342643       identity store
secrets-gen/    secrets-gen    secrets-gen_c4e68857    n/a
sys/            system         system_8e67315d         system endpoints used for control, policy and debugging

$ vault secrets disable secrets-gen

Dec 19 17:44:09 vaulthost vault[28883]: 2020-12-19T17:44:09.363Z [ERROR] secrets.secrets-gen.secrets-gen_c4e68857.secrets-gen.vault-secrets-gen: plugin tls init: error="error during token unwrap request: Error making API request.
Dec 19 17:44:09 vaulthost vault[28883]: URL: PUT https://a.b.c.d:8200/v1/sys/wrapping/unwrap
Dec 19 17:44:09 vaulthost vault[28883]: Code: 403. Errors:
Dec 19 17:44:09 vaulthost vault[28883]: * permission denied" timestamp=2020-12-19T17:44:09.362Z
Dec 19 17:44:09 vaulthost vault[28883]: 2020-12-19T17:44:09.364Z [ERROR] rollback: error rolling back: path=secrets-gen/ error="Unrecognized remote plugin message:

[voiprodrigo]

Ok, so it seems that in order to unmount the path I need to deregister the plugin and then restart the cluster. Very unexpected, but it was the only way I got going.

So re-doing everything from scratch, as soon as I enable the path again, I get the 403 permission denied for the unwrap.

[voiprodrigo]

Looks related to hashicorp/vault#7176, but this is supposedly fixed well before 1.6.1.

[voiprodrigo]

So, I disabled the auto-unseal to avoid setting the unseal token as environment variable, and the plugin is now working as expected. So the Vault issue is not fixed. Closing.

[github-actions]

This issue has been automatically locked since there has not been any
recent activity after it was closed. Please open a new issue for
related bugs.