Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Database Panics when loading particular Grafana dashboard #20308

Closed
kpdecker opened this issue Dec 10, 2020 · 6 comments · Fixed by #20322
Closed

Database Panics when loading particular Grafana dashboard #20308

kpdecker opened this issue Dec 10, 2020 · 6 comments · Fixed by #20322
Assignees
@danxmoran
Labels
area/2.x OSS 2.0 related issues and PRs help wanted panic

Comments

@kpdecker
Copy link

Steps to reproduce:

Unclear what preconditions existed to get here, but loading the attached Grafana dashboard would cause Influx to panic consistently.

Expected behavior:

Error is returned to requestor with clear information on how to resolve.

Actual behavior:

Database panics and terminates

panic: unexpected kind: got "invalid" expected "object", trace: goroutine 78434 [running]:
runtime/debug.Stack(0x3a67960, 0xc007094bd0, 0x53b6970)
	/usr/local/go/src/runtime/debug/stack.go:24 +0x9f
github.com/influxdata/flux/values.UnexpectedKind(0x0, 0xb, 0x0, 0x0)
	/go/pkg/mod/github.com/influxdata/flux@v0.94.0/values/values.go:287 +0x34
github.com/influxdata/flux/values.null.Object(...)
	/go/pkg/mod/github.com/influxdata/flux@v0.94.0/values/values.go:316
github.com/influxdata/flux/compiler.(*memberEvaluator).Eval(0xc00c92c500, 0x3a42520, 0xc00708b9e0, 0x3a69be0, 0xc007087b80, 0x40edb0, 0xc0070d34e0, 0x20, 0x18)
	/go/pkg/mod/github.com/influxdata/flux@v0.94.0/compiler/runtime.go:501 +0xab
github.com/influxdata/flux/compiler.eval(...)
	/go/pkg/mod/github.com/influxdata/flux@v0.94.0/compiler/runtime.go:86
github.com/influxdata/flux/compiler.(*objEvaluator).Eval.func1(0xc0070d34e0, 0xc0070d34e0, 0x0)
	/go/pkg/mod/github.com/influxdata/flux@v0.94.0/compiler/runtime.go:209 +0x15e
github.com/influxdata/flux/values.BuildObjectWithSize(0x0, 0xc0117f8908, 0x1, 0x3a18de0, 0xc0117f8988, 0x40edb0)
	/go/pkg/mod/github.com/influxdata/flux@v0.94.0/values/object.go:114 +0xcf
github.com/influxdata/flux/values.BuildObject(...)
	/go/pkg/mod/github.com/influxdata/flux@v0.94.0/values/object.go:101
github.com/influxdata/flux/compiler.(*objEvaluator).Eval(0xc0070949c0, 0x3a42520, 0xc00708b9e0, 0x3a69be0, 0xc007087b80, 0x605e040, 0x0, 0x4a40d8, 0xc0117f89f0)
	/go/pkg/mod/github.com/influxdata/flux@v0.94.0/compiler/runtime.go:197 +0x97
github.com/influxdata/flux/compiler.(*callEvaluator).Eval(0xc00c92c540, 0x3a42520, 0xc00708b9e0, 0x3a69be0, 0xc007087b80, 0xc006f7d680, 0xc0117f8a28, 0x8c49fa, 0x2385980)
	/go/pkg/mod/github.com/influxdata/flux@v0.94.0/compiler/runtime.go:543 +0x6c
github.com/influxdata/flux/compiler.(*logicalEvaluator).Eval(0xc007094ba0, 0x3a42520, 0xc00708b9e0, 0x3a69be0, 0xc007087b80, 0xc007087b80, 0x53b6970, 0x1, 0x7f40aec2dcf8)
	/go/pkg/mod/github.com/influxdata/flux@v0.94.0/compiler/runtime.go:250 +0x6f
github.com/influxdata/flux/compiler.eval(...)
	/go/pkg/mod/github.com/influxdata/flux@v0.94.0/compiler/runtime.go:86
github.com/influxdata/flux/compiler.(*blockEvaluator).Eval(0xc00c92c600, 0x3a42520, 0xc00708b9e0, 0x3a69be0, 0xc007087b80, 0xc007082e60, 0x0, 0x0, 0x40b85f)
	/go/pkg/mod/github.com/influxdata/flux@v0.94.0/compiler/runtime.go:106 +0x9a
github.com/influxdata/flux/compiler.eval(...)
	/go/pkg/mod/github.com/influxdata/flux@v0.94.0/compiler/runtime.go:86
github.com/influxdata/flux/compiler.compiledFn.Eval(0x3a18460, 0xc00c92c600, 0x3a69be0, 0xc007087b80, 0x3a42520, 0xc00708b9e0, 0x3a6fc40, 0xc007082e60, 0x25b44e0, 0xc007087b90, ...)
	/go/pkg/mod/github.com/influxdata/flux@v0.94.0/compiler/runtime.go:50 +0xb9
github.com/influxdata/flux/execute.(*RowPredicatePreparedFn).Eval(0xc00c92c680, 0x3a42520, 0xc00708b9e0, 0x3a6fc40, 0xc007082eb0, 0xc007094c60, 0x4c2276, 0x25fa060)
	/go/pkg/mod/github.com/influxdata/flux@v0.94.0/execute/row_fn.go:247 +0xe7
github.com/influxdata/flux/stdlib/universe.(*filterTransformation).filter(0xc00a41d4c0, 0xc00c92c680, 0x3a67c60, 0xc0070828c0, 0x3a6fc40, 0xc007082eb0, 0xc004827c60, 0x2, 0x2, 0x8, ...)
	/go/pkg/mod/github.com/influxdata/flux@v0.94.0/stdlib/universe/filter.go:315 +0x233
github.com/influxdata/flux/stdlib/universe.(*filterTransformation).filterTable.func1.1(0x3a67c60, 0xc0070828c0, 0x0, 0x0)
	/go/pkg/mod/github.com/influxdata/flux@v0.94.0/stdlib/universe/filter.go:280 +0xd3
github.com/influxdata/influxdb/v2/storage/flux.(*table).do(0xc00c0869c0, 0xc008ca8300, 0xc00884de98, 0x0, 0x0)
	/go/src/github.com/influxdata/influxdb/storage/flux/table.go:92 +0xb1
github.com/influxdata/influxdb/v2/storage/flux.(*floatTable).Do(0xc00c0869c0, 0xc008ca8300, 0x2, 0xc00a91c201)
	/go/src/github.com/influxdata/influxdb/storage/flux/table.gen.go:83 +0x5a
github.com/influxdata/flux/stdlib/universe.(*filterTransformation).filterTable.func1(0x3a42460, 0xc00c92c6c0, 0xc00c92c700, 0x0, 0xc008ebf3e0)
	/go/pkg/mod/github.com/influxdata/flux@v0.94.0/stdlib/universe/filter.go:279 +0x112
github.com/influxdata/flux/internal/execute/table.StreamWithContext.func1(0xc00b76b020, 0xc00b76afc0, 0xc007082f00, 0x3a42460, 0xc00c92c6c0, 0xc00c92c700)
	/go/pkg/mod/github.com/influxdata/flux@v0.94.0/internal/execute/table/stream.go:109 +0xb0
created by github.com/influxdata/flux/internal/execute/table.StreamWithContext
	/go/pkg/mod/github.com/influxdata/flux@v0.94.0/internal/execute/table/stream.go:106 +0x1b1

Environment info:

  • System info: Linux 4.15.0-126-generic x86_64
  • InfluxDB version: quay.io/influxdb/influxdb:v2.0.2
  • Other relevant environment details:
    Docker version 19.03.14, build 5eb3275d40
    Nomad running docker image
    cpu = 100
    memory = 4096

Config:

  env {
    INFLUXD_LOG_LEVEL = "debug"
  }
@kpdecker
Copy link
Author

kpdecker commented Dec 10, 2020 •
edited
Loading

Grafana Dashboard 
{
  "meta": {
    "type": "db",
    "canSave": false,
    "canEdit": false,
    "canAdmin": false,
    "canStar": true,
    "slug": "job-status",
    "url": "/d/FLxdu1AGk/job-status",
    "expires": "0001-01-01T00:00:00Z",
    "created": "2020-12-01T12:05:06Z",
    "updated": "2020-12-08T19:24:42Z",
    "updatedBy": "admin",
    "createdBy": "admin",
    "version": 12,
    "hasAcl": false,
    "isFolder": false,
    "folderId": 0,
    "folderTitle": "General",
    "folderUrl": "",
    "provisioned": false,
    "provisionedExternalId": ""
  },
  "dashboard": {
    "annotations": {
      "list": [
        {
          "$$hashKey": "object:288",
          "builtIn": 1,
          "datasource": "-- Grafana --",
          "enable": true,
          "hide": true,
          "iconColor": "rgba(0, 211, 255, 1)",
          "name": "Annotations & Alerts",
          "type": "dashboard"
        }
      ]
    },
    "editable": true,
    "gnetId": null,
    "graphTooltip": 1,
    "id": 3,
    "iteration": 1607455091813,
    "links": [
      {
        "icon": "external link",
        "includeVars": false,
        "keepTime": true,
        "tags": [
          "nomad"
        ],
        "targetBlank": true,
        "type": "dashboards"
      }
    ],
    "panels": [
      {
        "aliasColors": {},
        "bars": false,
        "dashLength": 10,
        "dashes": false,
        "datasource": null,
        "fieldConfig": {
          "defaults": {
            "custom": {}
          },
          "overrides": []
        },
        "fill": 1,
        "fillGradient": 0,
        "gridPos": {
          "h": 8,
          "w": 12,
          "x": 0,
          "y": 0
        },
        "hiddenSeries": false,
        "id": 18,
        "legend": {
          "avg": false,
          "current": false,
          "max": false,
          "min": false,
          "show": true,
          "total": false,
          "values": false
        },
        "lines": true,
        "linewidth": 1,
        "nullPointMode": "null",
        "options": {
          "alertThreshold": true
        },
        "percentage": false,
        "pluginVersion": "7.3.4",
        "pointradius": 2,
        "points": false,
        "renderer": "flot",
        "seriesOverrides": [],
        "spaceLength": 10,
        "stack": false,
        "steppedLine": false,
        "targets": [
          {
            "groupBy": [
              {
                "params": [
                  "$__interval"
                ],
                "type": "time"
              },
              {
                "params": [
                  "null"
                ],
                "type": "fill"
              }
            ],
            "orderByTime": "ASC",
            "policy": "default",
            "query": "filterVariable = (field, variable) => if variable == \"*\" then true else field == variable\n\nfrom(bucket: \"operations\")\n  |> range(start: v.timeRangeStart, stop: v.timeRangeStop)\n  |> filter(fn: (r) => r[\"_measurement\"] == \"coreMessageTypes\" and r[\"_field\"] == \"activebytes\"\n        and filterVariable(field: r.nomad.job, variable: \"${Job}\")\n        and filterVariable(field: r.host, variable: \"${Host}\"))\n  |> group(columns: [\"host\", \"nomad.job\", \"messagetype\"])\n  |> aggregateWindow(every: v.windowPeriod, fn: max, createEmpty: false)\n  |> yield(name: \"max\")",
            "refId": "A",
            "resultFormat": "time_series",
            "select": [
              [
                {
                  "params": [
                    "value"
                  ],
                  "type": "field"
                },
                {
                  "params": [],
                  "type": "mean"
                }
              ]
            ],
            "tags": []
          }
        ],
        "thresholds": [],
        "timeFrom": null,
        "timeRegions": [],
        "timeShift": null,
        "title": "Panel Title",
        "tooltip": {
          "shared": true,
          "sort": 0,
          "value_type": "individual"
        },
        "type": "graph",
        "xaxis": {
          "buckets": null,
          "mode": "time",
          "name": null,
          "show": true,
          "values": []
        },
        "yaxes": [
          {
            "$$hashKey": "object:564",
            "format": "short",
            "label": null,
            "logBase": 1,
            "max": null,
            "min": null,
            "show": true
          },
          {
            "$$hashKey": "object:565",
            "format": "short",
            "label": null,
            "logBase": 1,
            "max": null,
            "min": null,
            "show": true
          }
        ],
        "yaxis": {
          "align": false,
          "alignLevel": null
        }
      },
      {
        "datasource": "Winbeats",
        "fieldConfig": {
          "defaults": {
            "custom": {}
          },
          "overrides": []
        },
        "gridPos": {
          "h": 8,
          "w": 12,
          "x": 12,
          "y": 0
        },
        "id": 16,
        "options": {
          "showLabels": false,
          "showTime": true,
          "sortOrder": "Descending",
          "wrapLogMessage": true
        },
        "pluginVersion": "7.3.4",
        "targets": [
          {
            "bucketAggs": [
              {
                "$$hashKey": "object:374",
                "field": "@timestamp",
                "id": "2",
                "settings": {
                  "interval": "auto",
                  "min_doc_count": 0,
                  "trimEdges": 0
                },
                "type": "date_histogram"
              }
            ],
            "metrics": [
              {
                "$$hashKey": "object:343",
                "field": "select field",
                "id": "1",
                "meta": {},
                "settings": {},
                "type": "logs"
              }
            ],
            "query": "event.provider:$Job* AND host.name:$Host",
            "refId": "A",
            "timeField": "@timestamp"
          }
        ],
        "timeFrom": null,
        "timeShift": null,
        "title": "Windows Events",
        "type": "logs"
      },
      {
        "aliasColors": {},
        "bars": false,
        "dashLength": 10,
        "dashes": false,
        "datasource": null,
        "fieldConfig": {
          "defaults": {
            "custom": {},
            "unit": "percent"
          },
          "overrides": []
        },
        "fill": 1,
        "fillGradient": 0,
        "gridPos": {
          "h": 8,
          "w": 12,
          "x": 0,
          "y": 8
        },
        "hiddenSeries": false,
        "id": 12,
        "legend": {
          "avg": false,
          "current": false,
          "max": false,
          "min": false,
          "show": true,
          "total": false,
          "values": false
        },
        "lines": true,
        "linewidth": 1,
        "nullPointMode": "null",
        "options": {
          "alertThreshold": true
        },
        "percentage": false,
        "pluginVersion": "7.3.4",
        "pointradius": 2,
        "points": false,
        "renderer": "flot",
        "seriesOverrides": [],
        "spaceLength": 10,
        "stack": false,
        "steppedLine": false,
        "targets": [
          {
            "groupBy": [
              {
                "params": [
                  "$__interval"
                ],
                "type": "time"
              },
              {
                "params": [
                  "null"
                ],
                "type": "fill"
              }
            ],
            "orderByTime": "ASC",
            "policy": "default",
            "query": "filterVariable = (field, variable) => if variable == \"*\" then true else field == variable\n\nfrom(bucket: \"operations\")\n  |> range(start: v.timeRangeStart, stop: v.timeRangeStop)\n  |> filter(fn: (r) => r._measurement == \"nomad_client_allocs_cpu_total_percent\"\n        and filterVariable(field: r.job, variable: \"${Job}\")\n        and filterVariable(field: r.host, variable: \"${Host}\"))\n  |> keep(columns: [\"_time\", \"_value\", \"host\", \"job\"])\n  |> aggregateWindow(every: 30s, fn: max)\n  |> yield()\n",
            "refId": "A",
            "resultFormat": "time_series",
            "select": [
              [
                {
                  "params": [
                    "value"
                  ],
                  "type": "field"
                },
                {
                  "params": [],
                  "type": "mean"
                }
              ]
            ],
            "tags": []
          }
        ],
        "thresholds": [],
        "timeFrom": null,
        "timeRegions": [],
        "timeShift": null,
        "title": "% CPU Usage",
        "tooltip": {
          "shared": true,
          "sort": 0,
          "value_type": "individual"
        },
        "type": "graph",
        "xaxis": {
          "buckets": null,
          "mode": "time",
          "name": null,
          "show": true,
          "values": []
        },
        "yaxes": [
          {
            "$$hashKey": "object:85",
            "format": "percent",
            "label": null,
            "logBase": 1,
            "max": null,
            "min": null,
            "show": true
          },
          {
            "$$hashKey": "object:86",
            "format": "short",
            "label": null,
            "logBase": 1,
            "max": null,
            "min": null,
            "show": true
          }
        ],
        "yaxis": {
          "align": false,
          "alignLevel": null
        }
      },
      {
        "datasource": "Filebeats",
        "fieldConfig": {
          "defaults": {
            "custom": {
              "align": null,
              "filterable": false
            },
            "mappings": [],
            "thresholds": {
              "mode": "absolute",
              "steps": [
                {
                  "color": "green",
                  "value": null
                },
                {
                  "color": "red",
                  "value": 80
                }
              ]
            }
          },
          "overrides": []
        },
        "gridPos": {
          "h": 16,
          "w": 24,
          "x": 0,
          "y": 16
        },
        "id": 14,
        "options": {
          "showLabels": true,
          "showTime": true,
          "sortOrder": "Descending",
          "wrapLogMessage": true
        },
        "pluginVersion": "7.3.4",
        "repeat": null,
        "repeatDirection": "v",
        "targets": [
          {
            "bucketAggs": [
              {
                "$$hashKey": "object:208",
                "field": "@timestamp",
                "id": "2",
                "settings": {
                  "interval": "auto",
                  "min_doc_count": 0,
                  "trimEdges": 0
                },
                "type": "date_histogram"
              }
            ],
            "metrics": [
              {
                "$$hashKey": "object:206",
                "field": "select field",
                "id": "1",
                "meta": {},
                "settings": {},
                "type": "logs"
              }
            ],
            "query": "nomad.job:$Job AND agent.hostname:$Host",
            "refId": "A",
            "timeField": "@timestamp"
          }
        ],
        "timeFrom": null,
        "timeShift": null,
        "title": "Logs",
        "transformations": [
          {
            "id": "labelsToFields",
            "options": {}
          }
        ],
        "type": "logs"
      }
    ],
    "refresh": "",
    "schemaVersion": 26,
    "style": "dark",
    "tags": [],
    "templating": {
      "list": [
        {
          "allValue": "*",
          "current": {
            "selected": false,
            "text": "All",
            "value": "$__all"
          },
          "datasource": "Metrics",
          "definition": "",
          "error": null,
          "hide": 0,
          "includeAll": true,
          "label": null,
          "multi": false,
          "name": "Job",
          "options": [],
          "query": "from(bucket: \"operations\")\n  |> range(start: -1h)\n  |> filter(fn: (r) =>r._measurement == \"nomad_client_allocs_cpu_allocated\")\n  |> keep(columns: [\"job\"])\n  |> distinct()\n  |> map(fn: (r) => ({ _value: r.job }))",
          "refresh": 1,
          "regex": "",
          "skipUrlSync": false,
          "sort": 5,
          "tagValuesQuery": "",
          "tags": [],
          "tagsQuery": "",
          "type": "query",
          "useTags": false
        },
        {
          "allValue": "*",
          "current": {
            "selected": false,
            "text": "All",
            "value": "$__all"
          },
          "datasource": "Metrics",
          "definition": "",
          "error": null,
          "hide": 0,
          "includeAll": true,
          "label": null,
          "multi": false,
          "name": "Host",
          "options": [],
          "query": "filterVariable = (field, variable) => if variable == \"*\" then true else field == variable\n\nfrom(bucket: v.bucket)\n  |> range(start: -1h)\n  |> filter(fn: (r) => r._measurement == \"nomad_client_allocs_cpu_allocated\"\n      and filterVariable(field: r.job, variable: \"${Job}\"))\n  |> keep(columns: [\"host\"])\n  |> distinct()\n  |> map(fn: (r) => ({ _value: r.host }))",
          "refresh": 1,
          "regex": "",
          "skipUrlSync": false,
          "sort": 0,
          "tagValuesQuery": "",
          "tags": [],
          "tagsQuery": "",
          "type": "query",
          "useTags": false
        }
      ]
    },
    "time": {
      "from": "now-1h",
      "to": "now"
    },
    "timepicker": {
      "refresh_intervals": [
        "5s",
        "10s",
        "30s",
        "1m",
        "5m",
        "15m",
        "30m",
        "1h",
        "2h",
        "1d"
      ],
      "time_options": [
        "5m",
        "15m",
        "1h",
        "6h",
        "12h",
        "24h",
        "2d",
        "7d",
        "30d"
      ]
    },
    "timezone": "",
    "title": "Job Status",
    "uid": "FLxdu1AGk",
    "version": 12
  }
}

@timhallinflux
Copy link
Contributor

timhallinflux commented Dec 10, 2020 •
edited
Loading

does running this query inside the native UI also cause a panic?

from(bucket: "operations")
  |> range(start: -1h)  
  |> filter(fn: (r) => r._measurement == "nomad_client_allocs_cpu_allocated")
  |> keep(columns: ["job"])  
  |> distinct()
  |> map(fn: (r) => ({ _value: r.job }))

@danxmoran danxmoran added the area/2.x OSS 2.0 related issues and PRs label Dec 10, 2020
@danxmoran danxmoran self-assigned this Dec 10, 2020
@danxmoran
Copy link
Contributor

This might be influxdata/flux#3374

@danxmoran
Copy link
Contributor

I was able to reproduce this against 2.0.2 using the following (adjusted) query from the dashboard:

filterVariable = (field, variable) => 
  if variable == "*" then true 
  else field == variable
  
from(bucket: "telegraf/autogen")
  |> range(start: v.timeRangeStart, stop: v.timeRangeStop)
  |> filter(fn: (r) => r["_measurement"] == "cpu" and filterVariable(field: r.nomad.job, variable: "*") and filterVariable(field: r.host, variable: "*"))
  |> group(columns: ["host", "nomad.job", "messagetype"])
  |> aggregateWindow(every: v.windowPeriod, fn: max, createEmpty: false)
  |> yield(name: "max")

Will check if it's still failing on master.

@kpdecker
Copy link
Author

kpdecker commented Dec 10, 2020 via email

@danxmoran
Copy link
Contributor

This reproduces on influxd master for me BUT if I go get github.com/influxdb/flux@master, it seems to be fixed:
Screen Shot 2020-12-10 at 3 28 07 PM

@danxmoran danxmoran mentioned this issue Dec 10, 2020
Closed
@danxmoran danxmoran linked a pull request Dec 14, 2020 that will close this issue
build(flux): update Flux to v0.99.0 #20322
Merged
@danxmoran danxmoran mentioned this issue Dec 14, 2020
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@danxmoran danxmoran

Labels
area/2.x OSS 2.0 related issues and PRs help wanted panic
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

build(flux): update Flux to v0.99.0 influxdata/influxdb
3 participants
@kpdecker @danxmoran @timhallinflux