Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: updating telegraf plugins for latest release #20861

Merged
merged 9 commits into from
Mar 22, 2021
Merged
Show file tree
Hide file tree
Changes from 6 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
142 changes: 142 additions & 0 deletions ui/src/writeData/components/telegrafPlugins/aliyuncms.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,142 @@
# Alibaba (aka Aliyun) CloudMonitor Service Statistics Input
Here and after we use `Aliyun` instead `Alibaba` as it is default naming across web console and docs.

This plugin will pull Metric Statistics from Aliyun CMS.

### Aliyun Authentication

This plugin uses an [AccessKey](https://www.alibabacloud.com/help/doc-detail/53045.htm?spm=a2c63.p38356.b99.127.5cba21fdt5MJKr&parentId=28572) credential for Authentication with the Aliyun OpenAPI endpoint.
In the following order the plugin will attempt to authenticate.
1. Ram RoleARN credential if `access_key_id`, `access_key_secret`, `role_arn`, `role_session_name` is specified
2. AccessKey STS token credential if `access_key_id`, `access_key_secret`, `access_key_sts_token` is specified
3. AccessKey credential if `access_key_id`, `access_key_secret` is specified
4. Ecs Ram Role Credential if `role_name` is specified
5. RSA keypair credential if `private_key`, `public_key_id` is specified
6. Environment variables credential
7. Instance metadata credential

### Configuration:

```toml
## Aliyun Credentials
## Credentials are loaded in the following order
## 1) Ram RoleArn credential
## 2) AccessKey STS token credential
## 3) AccessKey credential
## 4) Ecs Ram Role credential
## 5) RSA keypair credential
## 6) Environment variables credential
## 7) Instance metadata credential

# access_key_id = ""
# access_key_secret = ""
# access_key_sts_token = ""
# role_arn = ""
# role_session_name = ""
# private_key = ""
# public_key_id = ""
# role_name = ""

# The minimum period for AliyunCMS metrics is 1 minute (60s). However not all
# metrics are made available to the 1 minute period. Some are collected at
# 3 minute, 5 minute, or larger intervals.
# See: https://help.aliyun.com/document_detail/51936.html?spm=a2c4g.11186623.2.18.2bc1750eeOw1Pv
# Note that if a period is configured that is smaller than the minimum for a
# particular metric, that metric will not be returned by the Aliyun OpenAPI
# and will not be collected by Telegraf.
#
## Requested AliyunCMS aggregation Period (required - must be a multiple of 60s)
period = "5m"

## Collection Delay (required - must account for metrics availability via AliyunCMS API)
delay = "1m"

## Recommended: use metric 'interval' that is a multiple of 'period' to avoid
## gaps or overlap in pulled data
interval = "5m"

## Metric Statistic Project (required)
project = "acs_slb_dashboard"

## Maximum requests per second, default value is 200
ratelimit = 200

## Discovery regions set the scope for object discovery, the discovered info can be used to enrich
## the metrics with objects attributes/tags. Discovery is supported not for all projects (if not supported, then
## it will be reported on the start - foo example for 'acs_cdn' project:
## 'E! [inputs.aliyuncms] Discovery tool is not activated: no discovery support for project "acs_cdn"' )
## Currently, discovery supported for the following projects:
## - acs_ecs_dashboard
## - acs_rds_dashboard
## - acs_slb_dashboard
## - acs_vpc_eip
##
## If not set, all regions would be covered, it can provide a significant load on API, so the recommendation here
## is to limit the list as much as possible. Allowed values: https://www.alibabacloud.com/help/zh/doc-detail/40654.htm
discovery_regions = ["cn-hongkong"]

## how often the discovery API call executed (default 1m)
#discovery_interval = "1m"

## Metrics to Pull (Required)
[[inputs.aliyuncms.metrics]]
## Metrics names to be requested,
## described here (per project): https://help.aliyun.com/document_detail/28619.html?spm=a2c4g.11186623.6.690.1938ad41wg8QSq
names = ["InstanceActiveConnection", "InstanceNewConnection"]

## Dimension filters for Metric (these are optional).
## This allows to get additional metric dimension. If dimension is not specified it can be returned or
## the data can be aggregated - it depends on particular metric, you can find details here: https://help.aliyun.com/document_detail/28619.html?spm=a2c4g.11186623.6.690.1938ad41wg8QSq
##
## Note, that by default dimension filter includes the list of discovered objects in scope (if discovery is enabled)
## Values specified here would be added into the list of discovered objects.
## You can specify either single dimension:
#dimensions = '{"instanceId": "p-example"}'

## Or you can specify several dimensions at once:
#dimensions = '[{"instanceId": "p-example"},{"instanceId": "q-example"}]'

## Enrichment tags, can be added from discovery (if supported)
## Notation is <measurement_tag_name>:<JMES query path (https://jmespath.org/tutorial.html)>
## To figure out which fields are available, consult the Describe<ObjectType> API per project.
## For example, for SLB: https://api.aliyun.com/#/?product=Slb&version=2014-05-15&api=DescribeLoadBalancers&params={}&tab=MOCK&lang=GO
#tag_query_path = [
# "address:Address",
# "name:LoadBalancerName",
# "cluster_owner:Tags.Tag[?TagKey=='cs.cluster.name'].TagValue | [0]"
# ]
## The following tags added by default: regionId (if discovery enabled), userId, instanceId.

## Allow metrics without discovery data, if discovery is enabled. If set to true, then metric without discovery
## data would be emitted, otherwise dropped. This cane be of help, in case debugging dimension filters, or partial coverage
## of discovery scope vs monitoring scope
#allow_dps_without_discovery = false
```

#### Requirements and Terminology

Plugin Configuration utilizes [preset metric items references](https://www.alibabacloud.com/help/doc-detail/28619.htm?spm=a2c63.p38356.a3.2.389f233d0kPJn0)

- `discovery_region` must be a valid Aliyun [Region](https://www.alibabacloud.com/help/doc-detail/40654.htm) value
- `period` must be a valid duration value
- `project` must be a preset project value
- `names` must be preset metric names
- `dimensions` must be preset dimension values

### Measurements & Fields:

Each Aliyun CMS Project monitored records a measurement with fields for each available Metric Statistic
Project and Metrics are represented in [snake case](https://en.wikipedia.org/wiki/Snake_case)

- aliyuncms_{project}
- {metric}_average (metric Average value)
- {metric}_minimum (metric Minimum value)
- {metric}_maximum (metric Maximum value)
- {metric}_value (metric Value value)

### Example Output:

```
$ ./telegraf --config telegraf.conf --input-filter aliyuncms --test
> aliyuncms_acs_slb_dashboard,instanceId=p-example,regionId=cn-hangzhou,userId=1234567890 latency_average=0.004810798017284538,latency_maximum=0.1100282669067383,latency_minimum=0.0006084442138671875
```
1 change: 0 additions & 1 deletion ui/src/writeData/components/telegrafPlugins/beat.md
Original file line number Diff line number Diff line change
@@ -1,7 +1,6 @@
# Beat Input Plugin
The Beat plugin will collect metrics from the given Beat instances. It is
known to work with Filebeat and Kafkabeat.

### Configuration:
```toml
## An URL from which to read beat-formatted JSON
Expand Down
3 changes: 3 additions & 0 deletions ui/src/writeData/components/telegrafPlugins/cloudwatch.md
Original file line number Diff line number Diff line change
Expand Up @@ -41,6 +41,9 @@ API endpoint. In the following order the plugin will attempt to authenticate.
## ex: endpoint_url = "http://localhost:8000"
# endpoint_url = ""

## Set http_proxy (telegraf uses the system wide proxy settings if it's is not set)
# http_proxy_url = "http://localhost:8888"

# The minimum period for Cloudwatch metrics is 1 minute (60s). However not all
# metrics are made available to the 1 minute period. Some are collected at
# 3 minute, 5 minute, or larger intervals. See https://aws.amazon.com/cloudwatch/faqs/#monitoring.
Expand Down
37 changes: 37 additions & 0 deletions ui/src/writeData/components/telegrafPlugins/csgo.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,37 @@
# Counter-Strike: Global Offensive (CSGO) Input Plugin

The `csgo` plugin gather metrics from Counter-Strike: Global Offensive servers.

#### Configuration
```toml
[[inputs.csgo]]
## Specify servers using the following format:
## servers = [
## ["ip1:port1", "rcon_password1"],
## ["ip2:port2", "rcon_password2"],
## ]
#
## If no servers are specified, no data will be collected
servers = []
```

### Metrics

The plugin retrieves the output of the `stats` command that is executed via rcon.

If no servers are specified, no data will be collected

- csgo
- tags:
- host
- fields:
- cpu (float)
- net_in (float)
- net_out (float)
- uptime_minutes (float)
- maps (float)
- fps (float)
- players (float)
- sv_ms (float)
- variance_ms (float)
- tick_ms (float)
3 changes: 3 additions & 0 deletions ui/src/writeData/components/telegrafPlugins/dcos.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,9 @@ your database.
options to exclude unneeded tags.
- Write to a database with an appropriate
[retention policy](https://docs.influxdata.com/influxdb/latest/guides/downsampling_and_retention/).
- Limit series cardinality in your database using the
[`max-series-per-database`](https://docs.influxdata.com/influxdb/latest/administration/config/#max-series-per-database-1000000) and
[`max-values-per-tag`](https://docs.influxdata.com/influxdb/latest/administration/config/#max-values-per-tag-100000) settings.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Needs reverted since these are gone in 2.0.4+

- Consider using the
[Time Series Index](https://docs.influxdata.com/influxdb/latest/concepts/time-series-index/).
- Monitor your databases
Expand Down
48 changes: 48 additions & 0 deletions ui/src/writeData/components/telegrafPlugins/directory_monitor.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,48 @@
# Directory Monitor Input Plugin

This plugin monitors a single directory (without looking at sub-directories), and takes in each file placed in the directory.
The plugin will gather all files in the directory at a configurable interval (`monitor_interval`), and parse the ones that haven't been picked up yet.

This plugin is intended to read files that are moved or copied to the monitored directory, and thus files should also not be used by another process or else they may fail to be gathered. Please be advised that this plugin pulls files directly after they've been in the directory for the length of the configurable `directory_duration_threshold`, and thus files should not be written 'live' to the monitored directory. If you absolutely must write files directly, they must be guaranteed to finish writing before the `directory_duration_threshold`.

### Configuration:

```toml
[[inputs.directory_monitor]]
## The directory to monitor and read files from.
directory = ""
#
## The directory to move finished files to.
finished_directory = ""
#
## The directory to move files to upon file error.
## If not provided, erroring files will stay in the monitored directory.
# error_directory = ""
#
## The amount of time a file is allowed to sit in the directory before it is picked up.
## This time can generally be low but if you choose to have a very large file written to the directory and it's potentially slow,
## set this higher so that the plugin will wait until the file is fully copied to the directory.
# directory_duration_threshold = "50ms"
#
## A list of the only file names to monitor, if necessary. Supports regex. If left blank, all files are ingested.
# files_to_monitor = ["^.*\.csv"]
#
## A list of files to ignore, if necessary. Supports regex.
# files_to_ignore = [".DS_Store"]
#
## Maximum lines of the file to process that have not yet be written by the
## output. For best throughput set to the size of the output's metric_buffer_limit.
## Warning: setting this number higher than the output's metric_buffer_limit can cause dropped metrics.
# max_buffered_metrics = 10000
#
## The maximum amount of file paths to queue up for processing at once, before waiting until files are processed to find more files.
## Lowering this value will result in *slightly* less memory use, with a potential sacrifice in speed efficiency, if absolutely necessary.
# file_queue_size = 100000
#
## The dataformat to be read from the files.
## Each data format has its own unique set of configuration options, read
## more about them here:
## https://github.com/influxdata/telegraf/blob/master/docs/DATA_FORMATS_INPUT.md
## NOTE: We currently only support parsing newline-delimited JSON. See the format here: https://github.com/ndjson/ndjson-spec
data_format = "influx"
```
25 changes: 21 additions & 4 deletions ui/src/writeData/components/telegrafPlugins/docker.md
Original file line number Diff line number Diff line change
Expand Up @@ -43,12 +43,29 @@ to gather stats from the [Engine API](https://docs.docker.com/engine/api/v1.24/)
## Timeout for docker list, info, and stats commands
timeout = "5s"

## Whether to report for each container per-device blkio (8:0, 8:1...) and
## network (eth0, eth1, ...) stats or not
## Whether to report for each container per-device blkio (8:0, 8:1...),
## network (eth0, eth1, ...) and cpu (cpu0, cpu1, ...) stats or not.
## Usage of this setting is discouraged since it will be deprecated in favor of 'perdevice_include'.
## Default value is 'true' for backwards compatibility, please set it to 'false' so that 'perdevice_include' setting
## is honored.
perdevice = true

## Whether to report for each container total blkio and network stats or not

## Specifies for which classes a per-device metric should be issued
## Possible values are 'cpu' (cpu0, cpu1, ...), 'blkio' (8:0, 8:1, ...) and 'network' (eth0, eth1, ...)
## Please note that this setting has no effect if 'perdevice' is set to 'true'
# perdevice_include = ["cpu"]

## Whether to report for each container total blkio and network stats or not.
## Usage of this setting is discouraged since it will be deprecated in favor of 'total_include'.
## Default value is 'false' for backwards compatibility, please set it to 'true' so that 'total_include' setting
## is honored.
total = false

## Specifies for which classes a total metric should be issued. Total is an aggregated of the 'perdevice' values.
## Possible values are 'cpu', 'blkio' and 'network'
## Total 'cpu' is reported directly by Docker daemon, and 'network' and 'blkio' totals are aggregated by this plugin.
## Please note that this setting has no effect if 'total' is set to 'false'
# total_include = ["cpu", "blkio", "network"]

## docker labels to include and exclude as tags. Globs accepted.
## Note that an empty array for both will include all labels as tags
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,8 @@ avoid cardinality issues:

- Use [metric filtering][] options to exclude unneeded measurements and tags.
- Write to a database with an appropriate [retention policy][].
- Limit series cardinality in your database using the
[max-series-per-database][] and [max-values-per-tag][] settings.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Needs reverted

- Consider using the [Time Series Index][tsi].
- Monitor your databases [series cardinality][].
- Consult the [InfluxDB documentation][influx-docs] for the most up-to-date techniques.
Expand All @@ -36,7 +38,7 @@ avoid cardinality issues:
```toml
[[inputs.kube_inventory]]
## URL for the Kubernetes API
url = "https://127.0.0.1"
url = "https://$HOSTIP:6443"

## Namespace to use. Set to "" to use all namespaces.
# namespace = "default"
Expand Down Expand Up @@ -307,6 +309,8 @@ kubernetes_statefulset,namespace=default,selector_select1=s1,statefulset_name=et

[metric filtering]: https://github.com/influxdata/telegraf/blob/master/docs/CONFIGURATION.md#metric-filtering
[retention policy]: https://docs.influxdata.com/influxdb/latest/guides/downsampling_and_retention/
[max-series-per-database]: https://docs.influxdata.com/influxdb/latest/administration/config/#max-series-per-database-1000000
[max-values-per-tag]: https://docs.influxdata.com/influxdb/latest/administration/config/#max-values-per-tag-100000
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Needs reverted

[tsi]: https://docs.influxdata.com/influxdb/latest/concepts/time-series-index/
[series cardinality]: https://docs.influxdata.com/influxdb/latest/query_language/spec/#show-cardinality
[influx-docs]: https://docs.influxdata.com/influxdb/latest/
Expand Down
4 changes: 4 additions & 0 deletions ui/src/writeData/components/telegrafPlugins/kubernetes.md
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,8 @@ avoid cardinality issues:

- Use [metric filtering][] options to exclude unneeded measurements and tags.
- Write to a database with an appropriate [retention policy][].
- Limit series cardinality in your database using the
[max-series-per-database][] and [max-values-per-tag][] settings.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Needs reverted

- Consider using the [Time Series Index][tsi].
- Monitor your databases [series cardinality][].
- Consult the [InfluxDB documentation][influx-docs] for the most up-to-date techniques.
Expand Down Expand Up @@ -152,6 +154,8 @@ kubernetes_system_container

[metric filtering]: https://github.com/influxdata/telegraf/blob/master/docs/CONFIGURATION.md#metric-filtering
[retention policy]: https://docs.influxdata.com/influxdb/latest/guides/downsampling_and_retention/
[max-series-per-database]: https://docs.influxdata.com/influxdb/latest/administration/config/#max-series-per-database-1000000
[max-values-per-tag]: https://docs.influxdata.com/influxdb/latest/administration/config/#max-values-per-tag-100000
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Needs reverted

[tsi]: https://docs.influxdata.com/influxdb/latest/concepts/time-series-index/
[series cardinality]: https://docs.influxdata.com/influxdb/latest/query_language/spec/#show-cardinality
[influx-docs]: https://docs.influxdata.com/influxdb/latest/
Expand Down
Loading