Vault crashes after unseal due to uninitialised tokenStore when using a custom plug-in #3241

kirilmonzo · 2017-08-25T19:12:06Z

Environment:
Cluster of 3 nodes running on Ubuntu 14.04 with Cassandra storage backend.

Vault Version: Vault v0.8.1 ('a7105536d613c0dce40d5439ae88fe0c5271298e') (Built from master, but the same issue occurred with the "stock" Vault v0.8.1 ('8d76a41854608c547a233f2e6292ae5355154695') )
Operating System/Architecture:
Ubuntu 14.04 - Linux host 3.13.0-74-generic #118-Ubuntu SMP Thu Dec 17 22:52:10 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux

Vault Config File:

backend "cassandra" {
  hosts = "cassandra-ZZZ"
  protocol_version = 3
  connection_timeout = 600
}

ha_storage "etcd" {
  address = "http://XXX:2379"
  ha_enabled = 1
  etcd_api = "v3"
  redirect_addr = "http://YYY:8200"
}

listener "tcp" {
  address     = "0.0.0.0:8200"
  tls_disable = 1
}

plugin_directory = "/opt/vault/plugins"

Startup Log Output:

==> Vault server configuration:

              HA Storage: etcd
                     Cgo: disabled
         Cluster Address: YYY:8201
              Listener 1: tcp (addr: "0.0.0.0:8200", cluster address: "0.0.0.0:8201", tls: "disabled")
               Log Level: trace
                   Mlock: supported: true, enabled: true
        Redirect Address: YYY:8200
                 Storage: cassandra
                 Version: Vault v0.8.1
             Version Sha: a7105536d613c0dce40d5439ae88fe0c5271298e

==> Vault server started! Log data will stream in below:

[TRACE] physical/cache: creating LRU cache: size=32768
[WARN ] Failed to dial %s: %v; please retry. IP:2379 grpc: the connection is closing
[WARN ] Failed to dial %s: %v; please retry. IP:2379 grpc: the connection is closing
[WARN ] Failed to dial %s: %v; please retry. IP:2379 grpc: the connection is closing
[WARN ] Failed to dial %s: %v; please retry. IP:2379 grpc: the connection is closing
[WARN ] Failed to dial %s: %v; please retry. XXX:2379 grpc: the connection is closing
[WARN ] Failed to dial %s: %v; please retry. IP:2379 grpc: the connection is closing
[TRACE] cluster listener addresses synthesized: cluster_addresses=[0.0.0.0:8201]
[DEBUG] core: cannot unseal, not enough keys: keys=1 threshold=3 nonce=AAAAAAAA-AAAA-AAAA-AAAA-AAAAAAAAAAAA
[DEBUG] core: cannot unseal, not enough keys: keys=2 threshold=3 nonce=AAAAAAAA-AAAA-AAAA-AAAA-AAAAAAAAAAAA
[INFO ] core: vault is unsealed
[INFO ] core: entering standby mode
[INFO ] core: acquired lock, enabling active operation
[TRACE] core: generating cluster private key
[TRACE] core: generating local cluster certificate
[INFO ] core: post-unseal setup starting
[TRACE] core: clearing forwarding clients
[TRACE] core: done clearing forwarding clients
[INFO ] core: loaded wrapping token key
[INFO ] core: successfully mounted backend: type=generic path=secret/
[INFO ] core: successfully mounted backend: type=system path=sys/
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x15f6bf1]

goroutine 128 [running]:
github.com/hashicorp/vault/vault.(*TokenStore).Salt(0x0, 0x0, 0x0, 0x0)
        /root/go/src/github.com/hashicorp/vault/vault/token_store.go:511 +0x41
github.com/hashicorp/vault/vault.(*TokenStore).SaltID(0x0, 0xc420392d80, 0x24, 0x0, 0x2, 0xed130d414, 0xc407c6ebfb)
        /root/go/src/github.com/hashicorp/vault/vault/token_store.go:631 +0x2f
github.com/hashicorp/vault/vault.(*TokenStore).create(0x0, 0xc4204ba8f0, 0x0, 0x0)
        /root/go/src/github.com/hashicorp/vault/vault/token_store.go:730 +0x126
github.com/hashicorp/vault/vault.(*Core).wrapInCubbyhole(0xc4201ad400, 0xc4205a5468, 0xc4205a5420, 0x7f06b8a52000, 0xc4205a5412, 0xc4205a54b8)
        /root/go/src/github.com/hashicorp/vault/vault/wrapping.go:111 +0x2cc
github.com/hashicorp/vault/vault.dynamicSystemView.ResponseWrapData(0xc4201ad400, 0xc42049f680, 0xc420392d50, 0xdf8475800, 0xc4201d4401, 0x28afa90, 0xc4205a5608, 0xc42019a060)
        /root/go/src/github.com/hashicorp/vault/vault/dynamic_system_view.go:113 +0x135
github.com/hashicorp/vault/vault.(*dynamicSystemView).ResponseWrapData(0xc420013f50, 0xc420392d50, 0xdf8475800, 0xc4201d4501, 0xc420392d50, 0x0, 0xc4201c52e0)
        <autogenerated>:162 +0x73
github.com/hashicorp/vault/helper/pluginutil.wrapServerConfig(0x7f06b8a1f7a8, 0xc420013f50, 0xc4204b0280, 0x24e, 0x24e, 0xc4204c8030, 0x0, 0x0, 0xc4201ac000, 0xc4203e1fb0)
        /root/go/src/github.com/hashicorp/vault/helper/pluginutil/tls.go:116 +0x229
github.com/hashicorp/vault/helper/pluginutil.(*PluginRunner).Run(0xc4204a1380, 0x7f06b8a1f7a8, 0xc420013f50, 0xc4203e1fb0, 0x1, 0x1bd81b3, 0x14, 0x1bfe49a, 0x24, 0xc4205a5840, ...)
        /root/go/src/github.com/hashicorp/vault/helper/pluginutil/runner.go:64 +0xfb
github.com/hashicorp/vault/logical/plugin.newPluginClient(0x7f06b8a1f7a8, 0xc420013f50, 0xc4204a1380, 0x285f1e0, 0xc42049b840, 0x0, 0x0, 0x100000000000000, 0x0)
        /root/go/src/github.com/hashicorp/vault/logical/plugin/plugin.go:82 +0x191
github.com/hashicorp/vault/logical/plugin.NewBackend(0xc4203f7540, 0x1d, 0x7f06b8a1f720, 0xc420013f50, 0x285f1e0, 0xc42049b840, 0x38, 0x38, 0x0, 0x0)
        /root/go/src/github.com/hashicorp/vault/logical/plugin/plugin.go:68 +0x2f2
github.com/hashicorp/vault/builtin/plugin.Backend(0xc4203ae700, 0xc4203e1ad0, 0x1bc2a74, 0xb, 0xc4201d4628)
        /root/go/src/github.com/hashicorp/vault/builtin/plugin/backend.go:37 +0xee
github.com/hashicorp/vault/builtin/plugin.Factory(0xc4203ae700, 0xc4203ae700, 0xc4204b9c96, 0x6, 0xc420067208)
        /root/go/src/github.com/hashicorp/vault/builtin/plugin/backend.go:19 +0x6f
github.com/hashicorp/vault/vault.(*Core).newLogicalBackend(0xc4201ad400, 0xc4204b9c96, 0x6, 0x285d8e0, 0xc420013f50, 0x28584e0, 0xc4203e1aa0, 0xc4203e1ad0, 0x2d, 0xc4204b9fb0, ...)
        /root/go/src/github.com/hashicorp/vault/vault/mount.go:768 +0x14d
github.com/hashicorp/vault/vault.(*Core).setupMounts(0xc4201ad400, 0x0, 0x0)
        /root/go/src/github.com/hashicorp/vault/vault/mount.go:686 +0x382
github.com/hashicorp/vault/vault.(*Core).postUnseal(0xc4201ad400, 0x0, 0x0)
        /root/go/src/github.com/hashicorp/vault/vault/core.go:1356 +0x42d
github.com/hashicorp/vault/vault.(*Core).runStandby(0xc4201ad400, 0xc4205986c0, 0xc420598720, 0xc420598780)
        /root/go/src/github.com/hashicorp/vault/vault/core.go:1573 +0x9e5
created by github.com/hashicorp/vault/vault.(*Core).unsealInternal
        /root/go/src/github.com/hashicorp/vault/vault/core.go:1000 +0x236

Expected Behavior:
Vault should successfully shutdown, unseal and load the plug-in. The plug-in worked after when it was loaded. Additionally if there are any problematic plugins they should be skipped during the initial mount load run, retried at the end and then eventually dropped.

Actual Behavior:
The Vault service crashes due to the tokenStore being nil at https://github.com/hashicorp/vault/blob/master/vault/wrapping.go#L111
From what I can tell this is because the setupMounts call that loads the plugin is before the setupCredentials call, which initialises the tokenStore.

Steps to Reproduce:
This issue is caused by a custom plugin. I can't share the whole plug-in, but it uses the same structure as the mock plugin inside Vault. As the error occurs during mount, rather than executing the APIs believe this should be okay. Please see source below:

main.go

import (
	"os"

	"log"

	"github.com/hashicorp/vault/helper/pluginutil"
	"github.com/hashicorp/vault/logical/plugin"
	customplugin "github.com/.../plugin"
)

func main() {
	apiClientMeta := &pluginutil.APIClientMeta{}
	flags := apiClientMeta.FlagSet()
	flags.Parse(os.Args)

	tlsConfig := apiClientMeta.GetTLSConfig()
	tlsProviderFunc := pluginutil.VaultPluginTLSProvider(tlsConfig)

	err := plugin.Serve(&plugin.ServeOpts{
		BackendFactoryFunc: customplugin.Factory,
		TLSProviderFunc:    tlsProviderFunc,
	})
	if err != nil {
		log.Println(err)
		os.Exit(1)
	}
}

plugin/backend.go

package plugin

import (
	"github.com/hashicorp/vault/logical"
	"github.com/hashicorp/vault/logical/framework"
)

func Factory(conf *logical.BackendConfig) (logical.Backend, error) {
	b := Backend()
	if err := b.Setup(conf); err != nil {
		return nil, err
	}
	return b, nil
}

func Backend() *backend {
	b := &backend{
		storage: &logical.InmemStorage{},
	}

	b.Backend = &framework.Backend{
		Help: "Help",
		Paths: []*framework.Path{
			pathX(b), // some custom APIs
		},
		Secrets:     []*framework.Secret{},
		BackendType: logical.TypeLogical,
	}

	return b
}

type backend struct {
	*framework.Backend

	storage *logical.InmemStorage
}

func pathX(b *backend) *framework.Path {
	return &framework.Path{
		Pattern: "test",
		Callbacks: map[logical.Operation]framework.OperationFunc{
		},
		HelpSynopsis:    "TODO",
		HelpDescription: "TODO",
	}
}

DO NOT attempt this on your production cluster, as it will brick it!

Setup Vault in non-dev mode
Compile the source code, go build -o myplugin ./main.go
Create plugin directory and add the plugin binary to it, e.g. /opt/vault/plugins
Add plugin_directory to your vault.conf/hcl, e.g. plugin_directory = "/opt/vault/plugins"
Run vault (non-dev mode!)
Unseal vault
Add the plugin to the plugin catalog, e.g.
vault write sys/plugins/catalog/myplugin sha_256=$(shasum -a 256 myplugin|cut -d ' ' -f 1) command="myplugin" executed in /opt/vault/plugins
Mount the plugin, vault mount -path=myplugir -plugin-name=myplugin plugin
Seal and restart Vault
Start Vault and unseal it

Important Factoids:
The plugin has BackendType = logical.TypeLogical. This crash leads to an irrecoverable failure as it's impossible to unseal Vault and remove the plugin.
This issue does not occur when Vault is ran in dev mode, as it's always unsealed.

The text was updated successfully, but these errors were encountered:

kirilmonzo · 2017-08-25T19:20:44Z

I'm unsure if this is going in the right direction of fixing it or just moving the problem further down:
Modifying setupMounts to either load plugins or nonplugins and pushing the plugins to be mounted after the setupExpiration call in core.go has unblocked this with the next error being plugin tls init which again creates an infinite load/fail/load loop.

The relevant error logs are

[DEBUG] plugin: starting plugin: path=/opt/vault/plugins/myplugin args=[/opt/vault/plugins/myplugin]
[DEBUG] plugin: waiting for RPC address: path=/opt/vault/plugins/myplugin
[ERROR] plugin.myplugin: plugin tls init: error=map[Outer:map[] Inner:map[]]
[DEBUG] plugin: plugin process exited: path=/opt/vault/plugins/myplugin
[ERROR] core: failed to create mount entry: path=myplugin/ error=plugin exited before we could connect
[INFO ] core: pre-seal teardown starting
[INFO ] core: cluster listeners not running
[INFO ] rollback: stopping rollback manager
[INFO ] core: pre-seal teardown complete
[ERROR] core: post-unseal setup failed: error=failed to setup mount table

jefferai · 2017-08-25T19:25:12Z

This might be as simple as moving the credential mount loading to happen before normal mount loading. Haven't thought through all of the possible consequences yet, though.

jefferai · 2017-08-25T19:25:29Z

@calvn throwing this your way

kirilmonzo · 2017-08-25T19:28:14Z

@jefferai - that's sort of what I did, but this leads to another irrecoverable situation due to the plugin not being able to initialise its TLS rpc connection I believe. In any case I think the plugin loading mechanism should probably be a bit more robust so that a failure to mount or execute the plugin doesn't break Vault. One thing I didn't mention is that this seemed to break the whole three node setup, as the mount configuration is shared among the nodes.

jefferai · 2017-08-25T20:26:29Z

@kirilmonzo Obviously we do not want the plugin loading mechanism to break Vault. Bugs exist though :-)

kirilmonzo · 2017-08-25T22:05:46Z

@jefferai Absolutely! Just wanted to make sure that I haven't left that bit out. Also really appreciate you jumping on this so quickly - please feel free to give me a shout if you need any more details. :-)

calvn · 2017-08-25T22:32:01Z

I was able to reproduce the panic and got a fix underway (by basically loading secret and credential plugin backends after c.setupExpiration()). However, I was not able to reproduce the plugin tls init error after applying those changes. I should have a branch for you to test it out soon.

kirilmonzo · 2017-08-26T07:03:38Z

Hi @calvn - Thanks! I think the plugin tls init error might be due to running the server with tls_disable. Would you be able to try that in case you haven't already?

calvn · 2017-08-26T15:55:28Z

I am able to reproduce the plugin tls init error, but haven't quite found a solution to that yet. I'll update this thread as soon as I get a working branch for you to test with.

calvn · 2017-08-26T16:14:21Z

I am testing with TLS enabled, but in my case I am getting http: TLS handshake error from 127.0.0.1:63423: remote error: tls: bad certificate

jefferai · 2017-08-26T16:35:41Z

If you're testing with TLS enabled but your cert is from an untrusted CA you probably need to add -tls-skip-verify to the plugin arfs.

If you're not using TLS you may need to add a custom -address? Possibly the address isn't being encoded in the token if there is no redirect_addr set on Vault.

calvn · 2017-08-28T19:56:25Z

@kirilmonzo I think I got a fix, but got to clean up the branch and include tests before opening the PR. If you wan to test it out on your end in the meantime, you can give the f-setup-plugins branch a try.

kirilmonzo · 2017-08-28T20:50:25Z

@calvn Thanks! I'll give it a go tomorrow and report back.

kirilmonzo · 2017-08-29T13:20:08Z

@calvn the version in the branch seems to fix the initial problem, but now there's another issue due to an error mounting the plugin. Believe this might be due to there being two mount entries for the same plugin/mountpoint. As this prevents unsealing Vault I'm not sure if it's possible to modify the entries in this version. Maybe some error handling for duplicate entries might help?

Logs below

==> Vault server configuration:

              HA Storage: etcd
                     Cgo: disabled
         Cluster Address: https://A:8201
              Listener 1: tcp (addr: "0.0.0.0:8200", cluster address: "0.0.0.0:8201", tls: "disabled")
               Log Level: trace
                   Mlock: supported: true, enabled: true
        Redirect Address: http://A:8200
                 Storage: cassandra
                 Version: Vault v0.8.1
             Version Sha: fc9601ffceaaa89af1e5d2f89a756fedf678f5c7+CHANGES

==> Vault server started! Log data will stream in below:

[TRACE] physical/cache: creating LRU cache: size=32768
[WARN ] Failed to dial %s: %v; please retry. IPADDR:2379 grpc: the connection is closing
[WARN ] Failed to dial %s: %v; please retry. IPADDR-2:2379 grpc: the connection is closing
[WARN ] Failed to dial %s: %v; please retry. IPADDR-3:2379 grpc: the connection is closing
[WARN ] Failed to dial %s: %v; please retry. IPADDR-2:2379 grpc: the connection is closing
[WARN ] Failed to dial %s: %v; please retry. etcd-host:2379 grpc: the connection is closing
[TRACE] cluster listener addresses synthesized: cluster_addresses=[0.0.0.0:8201]
[WARN ] grpc: addrConn.resetTransport failed to create client transport: connection error: desc = "transport: Error while dialing dial tcp IPADDR-3:2379: i/o timeout"; Reconnecting to {IPADDR-3:2379 <nil>}
[WARN ] Failed to dial %s: %v; please retry. IPADDR-3:2379 grpc: the connection is closing
[DEBUG] core: cannot unseal, not enough keys: keys=1 threshold=3 nonce=AAAAAAAA-AAAA-AAAA-AAAA-AAAAAAAAAAAA
[DEBUG] core: cannot unseal, not enough keys: keys=2 threshold=3 nonce=AAAAAAAA-AAAA-AAAA-AAAA-AAAAAAAAAAAA
[INFO ] core: vault is unsealed
[INFO ] core: entering standby mode
[INFO ] core: acquired lock, enabling active operation
[TRACE] core: generating cluster private key
[TRACE] core: generating local cluster certificate
[INFO ] core: post-unseal setup starting
[TRACE] core: clearing forwarding clients
[TRACE] core: done clearing forwarding clients
[INFO ] core: loaded wrapping token key
[INFO ] core: successfully setup plugin catalog: plugin-directory=/opt/vault/plugins
[INFO ] core: successfully mounted backend: type=system path=sys/
[INFO ] core: successfully mounted backend: type=cubbyhole path=cubbyhole/
[INFO ] rollback: starting rollback manager
[INFO ] expiration: restoring leases
[DEBUG] expiration: collecting leases
[DEBUG] expiration: leases collected: num_existing=6
[TRACE] expiration: leases loading: progress=0
[INFO ] expire: leases restored: restored_lease_count=6
[INFO ] core: successfully mounted backend: type=plugin path=myplugin/
[ERROR] core: failed to mount entry: path=myplugin/ error=cannot mount under existing mount 'myplugin/'
[INFO ] core: pre-seal teardown starting
[INFO ] core: cluster listeners not running
[INFO ] rollback: stopping rollback manager
[INFO ] core: pre-seal teardown complete
[ERROR] core: post-unseal setup failed: error=failed to setup mount table
[INFO ] core: acquired lock, enabling active operation
[TRACE] core: generating cluster private key
[TRACE] core: generating local cluster certificate
[INFO ] core: post-unseal setup starting
[TRACE] core: clearing forwarding clients
[TRACE] core: done clearing forwarding clients
[INFO ] core: loaded wrapping token key
[INFO ] core: successfully setup plugin catalog: plugin-directory=/opt/vault/plugins
[INFO ] core: successfully mounted backend: type=system path=sys/
[INFO ] core: successfully mounted backend: type=cubbyhole path=cubbyhole/
[INFO ] rollback: starting rollback manager
[INFO ] expiration: restoring leases
[DEBUG] expiration: collecting leases
[DEBUG] expiration: leases collected: num_existing=6
[TRACE] expiration: leases loading: progress=0
[INFO ] expire: leases restored: restored_lease_count=6
[INFO ] core: successfully mounted backend: type=plugin path=myplugin/
[ERROR] core: failed to mount entry: path=myplugin/ error=cannot mount under existing mount 'myplugin/'
[INFO ] core: pre-seal teardown starting
[INFO ] core: cluster listeners not running
[INFO ] rollback: stopping rollback manager
[INFO ] core: pre-seal teardown complete
[ERROR] core: post-unseal setup failed: error=failed to setup mount table
[INFO ] core: acquired lock, enabling active operation
[TRACE] core: generating cluster private key
... <repeats>

Additionally there might be an issue in one of the vendor'd dependencies. Seems to be fine after commenting out the test.

~/go/src/github.com/hashicorp/vault# make
==> Checking that code complies with gofmt requirements...
go generate
==> Removing old directory...
==> Building...
Number of parallel builds: 1

-->     linux/amd64: github.com/hashicorp/vault

1 errors occurred:
--> linux/amd64 error: exit status 2
Stderr: # github.com/hashicorp/vault/vault
vault/testing.go:588: t.Helper undefined (type "github.com/hashicorp/vault/vendor/github.com/mitchellh/go-testing-interface".T has no field or method Helper)

make: *** [dev] Error 1

kirilmonzo · 2017-08-29T13:31:33Z

Not returning error here https://github.com/hashicorp/vault/blob/f-setup-plugins/vault/mount.go#L777 seems to fix this, but unsure if it just masks a different problem, e.g. does that make the table consistent or just masks the inconsistency?

calvn · 2017-08-29T13:48:34Z

How did you get it to a state where it had two mounts at the same endpoint when sealed? Regarding the build error on t.Helper, that's a new method from the testing package on Go 1.9 so you'll have to build from that.

kirilmonzo · 2017-08-29T13:58:14Z

@calvn believe it may have happened whilst trying a couple of fixes locally, where I disabled plugin loading entirely, but that may have not properly updated the mounts table. What's a good way to delete the offending entries in the mounts table?

Output from executing unmount locally:

# vault mounts
Path            Type       Accessor            Plugin  Default TTL  Max TTL  Force No Cache  Replication Behavior  Description
...
myplugin/    plugin     plugin_6ad33d93     myplugin   system       system   false           replicated
...
# vault unmount myplugin
Successfully unmounted 'myplugin' if it was mounted
# vault mounts
Path            Type       Accessor            Plugin  Default TTL  Max TTL  Force No Cache  Replication Behavior  Description
...
myplugin/    plugin     plugin_6ad33d93     myplugin    system       system   false           replicated
...

Logs:

[DEBUG] plugin: plugin process exited: path=/opt/vault/plugins/myplugin
[WARN ] plugin: error closing client during Kill: err=connection is shut down
[INFO ] core: successfully unmounted: path=myplugin/

Cool, I'll up my local version to 1.9 :-)

calvn · 2017-08-29T15:53:25Z

I am not quite sure why the entry still exists after unmount, maybe @jefferai could chime in on this.

jefferai · 2017-08-29T17:23:25Z

Without knowing what the various fixes were that were tried locally I really can't comment other than to say that normally this is not an issue that should be encountered.

kirilmonzo · 2017-08-30T09:00:12Z

@jefferai I believe this is due to excluding the plugins in the setupMounts call. This probably has lead to a retry as the table was not modified. Is there an API I can use to edit the table directly? If not, that's fine. If there's time I think it's still worth making this a bit more resilient, i.e. an attempt to mount on top of an existing mount should not lead to an infinite loop in the unseal routine.

@calvn the fix in the branch seems to resolve the initially reported issue. About to reinitialise the Vault nodes and see if I hit any other problems. Thanks for your help!

Separate to this issue, think I may have spotted another small issue and was wondering if you can reproduce it - if a non-existing plugin is mounted (e.g. type in the plugin name) it can't be unmounted. Let me know if it's more appropriate to report this is in a separate issue ticket.

Command output:

# vault mount -path=plugin-1 -plugin-name=notexistingplugin  plugin
Successfully mounted plugin 'notexistingplugin' at 'plugin-1'!
# vault unmount plugin-1
Unmount error: Error making API request.

URL: DELETE http://127.0.0.1:8200/v1/sys/mounts/plugin-1
Code: 400. Errors:

* no plugin found with name: notexistingplugin
# vault mounts
Path            Type       Accessor            Plugin                           Default TTL  Max TTL  Force No Cache  Replication Behavior  Description
...
plugin-1/       plugin     plugin_c941fc4e     notexistingplugin                system       system   false           replicated

Logs:

[INFO ] core: successful mount: path=plugin-1/ type=plugin
[TRACE] rollback: attempting rollback: path=plugin-1/
[ERROR] rollback: error rolling back: path=plugin-1/ error=no plugin found with name: notexistingplugin
[TRACE] rollback: attempting rollback: path=plugin-1/
[ERROR] rollback: error rolling back: path=plugin-1/ error=no plugin found with name: notexistingplugin
[ERROR] sys: unmount failed: path=plugin-1/ error=no plugin found with name: notexistingplugin
[TRACE] rollback: attempting rollback: path=plugin-1/
[ERROR] rollback: error rolling back: path=plugin-1/ error=no plugin found with name: notexistingplugin

jefferai · 2017-08-30T16:32:55Z

Hi -- can you open a separate ticket for that issue? Thanks!

calvn · 2017-08-30T16:40:15Z

@kirilmonzo We realized that the current changes on branch is only a partial fix. Currently, it only works for secret plugin backends, since the dummy backend returned has logical.TypeLogical as its BackendType. The other problem is that special paths are not accessible before lazy loading occurs. We are working on addressing those issues, and will update you once we've got those changes in place.

kirilmonzo · 2017-08-31T10:34:56Z

@calvn Thanks for investigating this further and working out a proper full fix!
I'm not currently using special paths, so all seems to be working fine. The only error in the logs is
[ERR] yamux: Failed to write header: write unix @->/tmp/plugin762234406: write: broken pipe after unsealing Vault, but that doesn't seem to affect the plugin's operation.

kirilmonzo · 2017-09-01T11:56:27Z

Hi @calvn, just tried deploying the latest version (4d53b7d) from #3255 and having some trouble mounting the plugin. Logs below:

[ERROR] plugin.metadata.myplugin-binary: plugin tls init: error=map[]
[ERROR] sys: mount failed: path=myplugin/ error=plugin exited before we could connect

This is with a fresh installation. The previous set up was using d542582 with the above mount executing successfully. If it helps think this was due to a change somewhere between 4d53b7d and bd75790.

calvn · 2017-09-01T13:52:40Z

Did you rebuild your plugin binary? And are you using pluginutil.VaultPluginTLSProvider in there?

jefferai · 2017-09-01T14:06:31Z

@kirilmonzo also please pull from Master in case you didn't get a plugin version bump PR in your copy that came in pretty late. And as @calvn said you need to rebuild your plugin fresh.

jefferai · 2017-09-01T14:10:16Z

@kirilmonzo Also please build from master in case you're building against the final version of the PR, because the plugin bump is only in master and it's necessary.

kirilmonzo · 2017-09-01T15:08:54Z

@calvn, @jefferai - thanks guys! Didn't realise the plugin API has changed, so was indeed using an existing build of my plugin and Vault built from the PR branch, rather than master. Seems to work fine now!

jefferai · 2017-09-01T15:27:11Z

Great! Re-closing :-)

kirilmonzo changed the title ~~Vault crashes after unseal due to incorrect mount order when using plug-in~~ Vault crashes after unseal due to uninitialised tokenStore when using a custom plug-in Aug 25, 2017

jefferai added this to the 0.8.2 milestone Aug 25, 2017

jefferai assigned calvn Aug 25, 2017

calvn mentioned this issue Aug 29, 2017

Lazy-load plugin mounts #3255

Merged

calvn closed this as completed Aug 30, 2017

calvn reopened this Aug 30, 2017

calvn closed this as completed in #3255 Sep 1, 2017

jefferai reopened this Sep 1, 2017

jefferai closed this as completed Sep 1, 2017

schulzh mentioned this issue Nov 22, 2017

Cannot unseal vault with wrong plugin checksum #3602

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Vault crashes after unseal due to uninitialised tokenStore when using a custom plug-in #3241

Vault crashes after unseal due to uninitialised tokenStore when using a custom plug-in #3241

kirilmonzo commented Aug 25, 2017 •

edited

Loading

kirilmonzo commented Aug 25, 2017 •

edited

Loading

jefferai commented Aug 25, 2017

jefferai commented Aug 25, 2017

kirilmonzo commented Aug 25, 2017 •

edited

Loading

jefferai commented Aug 25, 2017

kirilmonzo commented Aug 25, 2017

calvn commented Aug 25, 2017

kirilmonzo commented Aug 26, 2017

calvn commented Aug 26, 2017

calvn commented Aug 26, 2017

jefferai commented Aug 26, 2017

calvn commented Aug 28, 2017

kirilmonzo commented Aug 28, 2017

kirilmonzo commented Aug 29, 2017 •

edited

Loading

kirilmonzo commented Aug 29, 2017

calvn commented Aug 29, 2017

kirilmonzo commented Aug 29, 2017 •

edited

Loading

calvn commented Aug 29, 2017

jefferai commented Aug 29, 2017

kirilmonzo commented Aug 30, 2017 •

edited

Loading

jefferai commented Aug 30, 2017

calvn commented Aug 30, 2017

kirilmonzo commented Aug 31, 2017 •

edited

Loading

kirilmonzo commented Sep 1, 2017 •

edited

Loading

calvn commented Sep 1, 2017

jefferai commented Sep 1, 2017

jefferai commented Sep 1, 2017

kirilmonzo commented Sep 1, 2017

jefferai commented Sep 1, 2017

Vault crashes after unseal due to uninitialised tokenStore when using a custom plug-in #3241

Vault crashes after unseal due to uninitialised tokenStore when using a custom plug-in #3241

Comments

kirilmonzo commented Aug 25, 2017 • edited Loading

kirilmonzo commented Aug 25, 2017 • edited Loading

jefferai commented Aug 25, 2017

jefferai commented Aug 25, 2017

kirilmonzo commented Aug 25, 2017 • edited Loading

jefferai commented Aug 25, 2017

kirilmonzo commented Aug 25, 2017

calvn commented Aug 25, 2017

kirilmonzo commented Aug 26, 2017

calvn commented Aug 26, 2017

calvn commented Aug 26, 2017

jefferai commented Aug 26, 2017

calvn commented Aug 28, 2017

kirilmonzo commented Aug 28, 2017

kirilmonzo commented Aug 29, 2017 • edited Loading

kirilmonzo commented Aug 29, 2017

calvn commented Aug 29, 2017

kirilmonzo commented Aug 29, 2017 • edited Loading

calvn commented Aug 29, 2017

jefferai commented Aug 29, 2017

kirilmonzo commented Aug 30, 2017 • edited Loading

jefferai commented Aug 30, 2017

calvn commented Aug 30, 2017

kirilmonzo commented Aug 31, 2017 • edited Loading

kirilmonzo commented Sep 1, 2017 • edited Loading

calvn commented Sep 1, 2017

jefferai commented Sep 1, 2017

jefferai commented Sep 1, 2017

kirilmonzo commented Sep 1, 2017

jefferai commented Sep 1, 2017

kirilmonzo commented Aug 25, 2017 •

edited

Loading

kirilmonzo commented Aug 25, 2017 •

edited

Loading

kirilmonzo commented Aug 25, 2017 •

edited

Loading

kirilmonzo commented Aug 29, 2017 •

edited

Loading

kirilmonzo commented Aug 29, 2017 •

edited

Loading

kirilmonzo commented Aug 30, 2017 •

edited

Loading

kirilmonzo commented Aug 31, 2017 •

edited

Loading

kirilmonzo commented Sep 1, 2017 •

edited

Loading