Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

enterprise: Default Lease Count Quota #24382

Merged
merged 7 commits into from
Dec 8, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions changelog/24382.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
```release-note:feature
**Default Lease Count Quota**: Apply a new global default lease count quota of 300k leases for all
new installs of Vault.
```
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: given that lease count quotas are an enterprise only feature, does it make more sense to have the changelog be on the enterprise PR?

1 change: 1 addition & 0 deletions vault/core.go
Original file line number Diff line number Diff line change
Expand Up @@ -3494,6 +3494,7 @@ func (c *Core) setupQuotas(ctx context.Context, isPerfStandby bool) error {
qmFlags := &quotas.ManagerFlags{
IsPerfStandby: isPerfStandby,
IsDRSecondary: c.IsDRSecondary(),
IsNewInstall: c.IsNewInstall(ctx),
}

return c.quotaManager.Setup(ctx, c.systemBarrierView, qmFlags)
Expand Down
7 changes: 7 additions & 0 deletions vault/quotas/quotas.go
Original file line number Diff line number Diff line change
Expand Up @@ -1147,6 +1147,7 @@ func Load(ctx context.Context, storage logical.Storage, qType, name string) (Quo
type ManagerFlags struct {
IsPerfStandby bool
IsDRSecondary bool
IsNewInstall bool
}

// Setup loads the quota configuration and all the quota rules into the
Expand All @@ -1167,6 +1168,7 @@ func (m *Manager) Setup(ctx context.Context, storage logical.Storage, flags *Man
}
m.isPerfStandby = flags.IsPerfStandby
m.isDRSecondary = flags.IsDRSecondary
m.isNewInstall = flags.IsNewInstall

// Load the quota configuration from storage and load it into the quota
// manager.
Expand Down Expand Up @@ -1199,6 +1201,11 @@ func (m *Manager) Setup(ctx context.Context, storage logical.Storage, flags *Man
m.setEnableRateLimitAuditLoggingLocked(config.EnableRateLimitAuditLogging)
m.setEnableRateLimitResponseHeadersLocked(config.EnableRateLimitResponseHeaders)
m.setRateLimitExemptPathsLocked(exemptPaths)

if err := m.setupDefaultLeaseCountQuotaInStorage(ctx); err != nil {
m.logger.Warn("skipping initialization of default lease count quota", "error", err)
}

if err = m.resetCache(); err != nil {
return err
}
Expand Down
5 changes: 5 additions & 0 deletions vault/quotas/quotas_util.go
Original file line number Diff line number Diff line change
Expand Up @@ -32,9 +32,14 @@ func (m *Manager) inLeasePathCache(path string) bool {
return false
}

func (m *Manager) setupDefaultLeaseCountQuotaInStorage(_ctx context.Context) error {
return nil
}

type entManager struct {
isPerfStandby bool
isDRSecondary bool
isNewInstall bool
}

func (*entManager) Reset() error {
Expand Down
19 changes: 19 additions & 0 deletions vault/version_store.go
Original file line number Diff line number Diff line change
Expand Up @@ -182,6 +182,25 @@ func (c *Core) isMajorVersionFirstMount(ctx context.Context) bool {
return isMilestoneUpdate || isMajorVersionUpdate
}

// IsNewInstall consults the version store to check for empty history. This
// property should hold during unseal of new installations of Vault.
func (c *Core) IsNewInstall(ctx context.Context) bool {
oldestVersion, _, err := c.FindOldestVersionTimestamp()
if err != nil {
return false
}

newestVersion, _, err := c.FindNewestVersionTimestamp()
if err != nil {
return false
}

// We store the Vault version history at the end of unseal setup. During
// early unseal on a new install, the old and new versions will both be
// empty.
return oldestVersion == "" && newestVersion == ""
}

func IsJWT(token string) bool {
return len(token) > 3 && strings.Count(token, ".") == 2 &&
(token[3] != '.' && token[1] != '.')
Expand Down
52 changes: 52 additions & 0 deletions vault/version_store_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -72,6 +72,58 @@ func TestVersionStore_GetOldestVersion(t *testing.T) {
}
}

// TestVersionStore_IsNewInstall consults the version store to see if version
// history is empty. This property should hold during early unseal of a new
// Vault installation.
func TestVersionStore_IsNewInstall(t *testing.T) {
c, _, _ := TestCoreUnsealed(t)
now := time.Now().UTC()

// Remove version history to simulate early unseal
vaultVersionPath := "core/versions/"
key := vaultVersionPath + version.Version
if err := c.barrier.Delete(context.Background(), key); err != nil {
t.Fatal(err)
}

// delete the version from the map as well
delete(c.versionHistory, version.Version)

if newInstall := c.IsNewInstall(c.activeContext); !newInstall {
t.Fatal("expected IsNewInstall to return 'true', but got 'false'")
}

firstEntry := &VaultVersion{Version: "1.16.0", TimestampInstalled: now}
if _, err := c.storeVersionEntry(context.Background(), firstEntry, false); err != nil {
t.Fatalf("failed to write version entry %#v, err: %s", firstEntry, err.Error())
}

if err := c.loadVersionHistory(c.activeContext); err != nil {
t.Fatalf("failed to populate version history cache, err: %s", err.Error())
}

if len(c.versionHistory) != 1 {
t.Fatalf("expected 1 entry in timestamps map after refresh, found: %d", len(c.versionHistory))
}
secondEntry := &VaultVersion{Version: "1.13.0", TimestampInstalled: now}
_, err := c.storeVersionEntry(context.Background(), secondEntry, false)
if err != nil {
t.Fatalf("failed to write version entry %#v, err: %s", secondEntry, err.Error())
}

err = c.loadVersionHistory(c.activeContext)
if err != nil {
t.Fatalf("failed to populate version history cache, err: %s", err.Error())
}

if len(c.versionHistory) != 2 {
t.Fatalf("expected 2 entry in timestamps map after refresh, found: %d", len(c.versionHistory))
}
if newInstall := c.IsNewInstall(c.activeContext); newInstall {
t.Fatal("expected IsNewInstall to return 'false', but got 'true'")
}
}

// TestVersionStore_GetNewestVersion verifies that FindNewestVersionTimestamp finds the newest
// (in time) vault version stored.
func TestVersionStore_GetNewestVersion(t *testing.T) {
Expand Down
105 changes: 71 additions & 34 deletions website/content/docs/enterprise/lease-count-quotas.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -11,43 +11,80 @@ description: |-

Vault features an extension to resource quotas that allows operators to enforce
limits on how many leases are created. For a given lease count quota, if the
number of leases in the cluster hits the configured limit, `max_leases`, additional
lease creations will be forbidden for all clients until a lease has been revoked
or has expired.
number of leases in the cluster hits the configured limit, `max_leases`,
additional lease creations will be forbidden for all clients until the
an operator modifies the configured limit, or a lease has been revoked or
expired.

## Root tokens

It is important to note that lease count quotas do not apply to the root tokens.
If the number of leases in the cluster hits the configured limit, `max_leases`,
an operator could still create a root token and access the cluster to try to recover.

Additionally, batch token creation is blocked when the lease count quota is
exceeded, but batch tokens do not count towards the quota.

All the nodes in the Vault cluster will share the lease quota rules, meaning that
the lease counters will be shared, regardless of which node in the Vault cluster
receives lease generation requests. Lease quotas can be imposed across Vault's API,
or scoped down to API pertaining to specific namespaces or specific mounts.

A quota that is defined in the `root` namespace with no specified path is inherited by all namespaces.
Essentially, it applies to the entire Vault API unless a more specific quota has been defined
for a specific API path.

Lease count quotas defined on a namespace take precedence over the global
quotas. Lease count quotas defined for a mount will take precedence over global
and namespace quotas. Lease count quotas defined for a specific path will take precedence
over global, namespace, and mount quotas. Lease count quotas defined with a login role for
a specific auth mount will take precedence over every other quota when applying to
login requests using that auth method and the specified role.

The limits on quotas can either be increased or decreased. If a lower precedence quota
is very restrictive and if it is desired to relax the limits in one namespace,
or on a specific mount, it can be done using this precedence model. On the
other hand, if a lower precedence quota is very liberal and if it is desired to
further restrict usages in a specific namespace or mount, that can be done
using the precedence model too.

Vault also allows the inspection into the state of lease count quotas in a Vault
cluster through various [metrics](/vault/docs/internals/telemetry/metrics/core-system#quota-metrics)
exposed and through enabling optional audit logging.
an operator can still create a root token and access the cluster to try to
recover.

## Batch tokens

Batch token creation is blocked when the lease count quota is exceeded, but
batch tokens do not count toward the quota.

All the nodes in the Vault cluster share the lease quota rules, meaning that the
lease counters are shared, regardless of which node in the Vault cluster
receives lease generation requests. Lease quotas can be imposed across Vault's
API, or scoped down to API pertaining to specific namespaces or specific mounts.

## Lease count quota inheritance

A quota that is defined in the `root` namespace with no specified path is
inherited by all namespaces. This type of quota is referred to as a `global`
quota. Global quotas applie to the entire Vault API unless a more specific quota
(higher precedence) quota has been defined.

## Lease count quota precedence

Lease count quota precedence is dictated by highest to lowest level of
specificity. The rules are as follows:

1. Global lease count quotas are applied to all mounts and namespaces only if no
other, more specific namespace is defined.
1. Lease count quotas defined on a namespace take precedence over the global
quotas.
1. Lease count quotas defined for a mount will take precedence over global
and namespace quotas.
1. Lease count quotas defined for a specific path will take
precedence over global, namespace, and mount quotas.
1. Lease count quotas defined with a login role for a specific auth mount will
take precedence over every other quota when applying to login requests using
that auth method and the specified role.

The limits on quotas can either be increased or decreased. If a lower precedence
quota is very restrictive and if it is desired to relax the limits in one
namespace, or on a specific mount, it can be done using this precedence model.
On the other hand, if a lower precedence quota is very liberal and if it is
desired to further restrict usages in a specific namespace or mount, that can be
done using the precedence model too.

## Default lease count quota

As of Vault 1.16.0, new installations of Vault Enterprise will include a default
global quota with a `max_leases` value of `100000`. This value is an
intentionally low limit, intended to prevent runaway leases in the event that no
other lease count quota is specified.

This limit will affect all new clusters with no pre-existing configuration. As
with any other quota, the default can be directly increased, decreased, or
removed using the [lease-count-quotas endpoints](/vault/api-docs/system/lease-count-quotas).

The default may also be overridden by higher precedence quotas (specified for a
namespace, mount, path, or role) as described in the [Lease count quota
precedence](#lease-count-quota-precedence) section above.

## Quota inspection

Vault also allows the inspection of the state of lease count quotas in a Vault
cluster through various
[metrics](/vault/docs/internals/telemetry/metrics/core-system#quota-metrics)
and through enabling optional audit logging.

## Tutorial

Expand Down