Installing and configuring Telegraf

Installing Telegraf

Installing Telegraf is straightforward on most Linux distributions with step-by-step instructions. On Ubuntu, for example:

# add the influxdata signing key
curl -sL https://repos.influxdata.com/influxdb.key | sudo apt-key add -

# configure a package repo
source /etc/lsb-release
echo "deb https://repos.influxdata.com/${DISTRIB_ID,,} ${DISTRIB_CODENAME} stable" | sudo tee /etc/apt/sources.list.d/influxdb.list

# install Telegraf and start the daemon
sudo apt-get update && sudo apt-get install telegraf
sudo systemctl enable telegraf
sudo systemctl start telegraf

Configuring Telegraf

Besides acting as a statsd agent, Telegraf can collect additional metrics of its own. Telegraf itself ships with a wide range of input plugins to collect data from lots of sources. We're going to enable some of the most common ones to monitor CPU, memory, disk I/O, networking, and process status.

The telegraf.conf file starts with global options:

[agent]
  interval = "10s"
  flush_interval = "10s"
  omit_hostname = false

We set the default collection interval to 10 seconds and ask Telegraf to include a host tag in each metric.

As mentioned above, Telegraf also allows you to set additional tags on the metrics that pass through it. In this case, we are adding tags for the server role and datacenter. We can then use these tags in Grafana to filter queries (for example, to create a dashboard showing only servers with the consul-server role, or only servers in the us-east-1 datacenter).

[global_tags]
  role = "consul-server"
  datacenter = "us-east-1"

Next, we set up a statsd listener on UDP port 8125, with instructions to calculate percentile metrics and to parse DogStatsd-compatible tags, when they're sent:

[[inputs.statsd]]
  protocol = "udp"
  service_address = ":8125"
  delete_gauges = true
  delete_counters = true
  delete_sets = true
  delete_timings = true
  percentiles = [90]
  metric_separator = "_"
  parse_data_dog_tags = true
  allowed_pending_messages = 10000
  percentile_limit = 1000

The full reference to all the available statsd-related options in Telegraf is here.

Now we can configure inputs for things like CPU, memory, network I/O, and disk I/O. Most of them don't require any configuration, but make sure the interfaces list in inputs.net matches the interface names you see in ifconfig.

[[inputs.cpu]]
  percpu = true
  totalcpu = true
  collect_cpu_time = false

[[inputs.disk]]
  # mount_points = ["/"]
  # ignore_fs = ["tmpfs", "devtmpfs"]

[[inputs.diskio]]
  # devices = ["sda", "sdb"]
  # skip_serial_number = false

[[inputs.kernel]]
  # no configuration

[[inputs.linux_sysctl_fs]]
  # no configuration

[[inputs.mem]]
  # no configuration

[[inputs.net]]
  interfaces = ["enp0s*"]

[[inputs.netstat]]
  # no configuration

[[inputs.processes]]
  # no configuration

[[inputs.procstat]]
  pattern = "(consul|vault)"

[[inputs.swap]]
  # no configuration

[[inputs.system]]
  # no configuration

Another useful plugin is the procstat plugin, which reports metrics for processes you select:

[[inputs.procstat]]
  pattern = "(consul|vault)"

Finally, Telegraf even includes a plugin to monitor Consul agents.

[[inputs.consul]]
  address = "localhost:8500"
  scheme = "http"

Putting it all together, you can see complete telegraf.conf examples for Consul and Vault hosts. While you're at it, you may as well set up monitoring on the InfluxDB/Grafana server too. Here's an example of how you could do that.

After you edit the telegraf.conf, don't forget to restart the Telegraf agent:

systemctl restart telegraf

Next up: Configuring Vault and Consul to send telemetry data to Telegraf.