Monitoring Vault server

[tamalsaha]

Telemetry: https://www.vaultproject.io/docs/internals/telemetry.html

Prometheus

• Telemetry: add prometheus endpoint option hashicorp/vault#2937
• https://github.com/grapeshot/vault_exporter

Tom Wilkie

• https://twitter.com/search?l=&q=statsd%20from%3Atom_wilkie&src=typd

Vault uses github.com/armon/go-metrics

coreos: https://coreos.com/tectonic/docs/latest/vault-operator/user/monitoring.html

[tamalsaha]

• Separate StatsService for monitoring

[sanjid133]

Local Setup

• Prometheus setup
• git clone git@github.com:prometheus/prometheus.git
• Run make build
• Create a file prom.yaml with following data:

global:
  scrape_interval:     15s # By default, scrape targets every 15 seconds.

  # Attach these labels to any time series or alerts when communicating with
  # external systems (federation, remote storage, Alertmanager).
  external_labels:
    monitor: 'codelab-monitor'

# A scrape configuration containing exactly one endpoint to scrape:
# Here it's Prometheus itself.
scrape_configs:
  # The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.
  - job_name: 'vault'

    # Override the global default and scrape targets from this job every 5 seconds.
    scrape_interval: 5s

    static_configs:
      - targets: ['localhost:9102']

• Run ./prometheus --config.file=prom.yaml

• Vault setup

• Install vault following https://www.vaultproject.io/docs/install/index.html

• Create a file vault.hcl with following data:

listener "tcp" {
  address = "127.0.0.1:8200"
  tls_disable = true
}

storage "inmem" {}

disable_mlock=true

ui=true

telemetry {
  statsd_address = "127.0.0.1:9125"
}

• Run vault server -config=vault.hcl

• Optionally you can run

 - vault operator init
 - vault operator unseal

• Setup Exporter

• Clone git@github.com:kubevault/vault_exporter.git

• Run go build

• Run ./prometheus_exporter

In browser http://localhost:9090

[tamalsaha]

• Update Statsd mapping file for remaining metrics.

[tamalsaha]

Grafana Dashboard:

• https://github.com/hashicorp/vault-consul-monitoring/blob/master/dashboards/vault_cluster_health.json
• https://grafana.com/dashboards/7700

[tamalsaha]

Done.