error creating dashboard, expected status 200 but got 404

[1337andre]

Hi,

i tried to import SimpleDashboard (https://github.com/integr8ly/grafana-operator/blob/master/deploy/examples/dashboards/SimpleDashboard.yaml)

$ kubectl apply -f https://github.com/integr8ly/grafana-operator/blob/master/deploy/examples/dashboards/SimpleDashboard.yaml

$ kubectl get grafanadashboards.integreatly.org -o yaml| grep -A 4 status:
  status:
    hash: ""
    id: 0
    message: error creating dashboard, expected status 200 but got 404
    phase: failing 

$ stern grafana-operator  | grep simple-dashboard                             
+ grafana-operator-5f8cb65798-hb2qp › grafana-operator                                
grafana-operator-5f8cb65798-hb2qp grafana-operator {"level":"info","ts":1581498721.864
8725,"logger":"controller_grafanadashboard","msg":"cannot submit dashboard monitoring/
simple-dashboard"} 

I use OLM to install grafana-operator in Version 3.0.2 in eks (k8s-version: v1.14.7-eks-1861c5)
Grafana was installed with auth.generic_oauth

apiVersion: integreatly.org/v1alpha1
kind: Grafana
metadata:
  name: example-grafana
spec:
  deployment:
    replicas: 2
    annotations:
      iam.amazonaws.com/role: k8s-cloudwatch-sts
  ingress:
    enabled: true
    annotations:
      cert-manager.io/cluster-issuer: letsencrypt-prod
      kubernetes.io/ingress.class: nginx
    hostname: grafana.eks.<my-domain>
    tlsEnabled: true
    tlsSecretName: grafana-eks-<my-domain>
  config:
    server:
      root_url: https://grafana.eks.<my-domain>
    log:
      mode: "console"
      level: "debug"
    security:
      admin_user: "<myuser>"
      admin_password: "<mypassword>"
    auth:
      disable_login_form: false
    auth.generic_oauth:
      enabled: true
      client_id: app-grafana
      role_attribute_path: roles[0]
      client_secret: <myclientsecret>
      scopes: openid profile email roles
      auth_url: https://idp.<my-domain>/auth/realms/<myrealm>/protocol/openid-connect/auth
      token_url: https://idp.<my-domain>/auth/realms/<myrealm>/protocol/openid-connect/token
      api_url: https://idp.<my-domain>/auth/realms/<myrealm>/protocol/openid-connect/userinfo
      allowed_domains: <my-domain>
    database:
      type: postgres
      host: grafana-cluster:5432
      name: userdb
      user: <myuser>
      password: <mypassword>
  dashboardLabelSelector:
    - matchExpressions:
        - {key: app, operator: In, values: [grafana]}

something I do wrong?

greetings
Andre

[pb82]

The CR seems fine at first glance. I'll give it a try later.

[cmoulliard]

The CR seems fine at first glance. I'll give it a try later.

My issue #137 is certainly related to this one as I see also as status error creating dashboard, expected status 200 but got 404

[cmoulliard]

When I debug, I observe that the HTTP Client created to access the Graphana instance is not correct. It is defined as such

req = {net/http.Request} POST https://88.99.186.195/api/dashboards/db
rawUrl = {string} "https://88.99.186.195/api/dashboards/db"

while in fact, it should be defined using either the kubernetes service + port or ingress route

Question: Is it possible to define the address of the service or that the operator calculate it ?

[pb82]

@cmoulliard The operator uses this strategy to determine the URL: https://github.com/integr8ly/grafana-operator/blob/master/pkg/controller/grafana/grafana_controller.go#L203

So it will first try to use the route/ingress if they exist. If not it will fall back to using the Service (more specifically: the clusterIP of the service). There is also a CR setting preferService to always override the route/ingress and use the service.

Are you running on vanilla Kubernetes or OpenShift?

[cmoulliard]

Are you running on vanilla Kubernetes or OpenShift?

Kubernetes with an ingress route

[cmoulliard]

So it will first try to use the route/ingress if they exist

As the ingress route exists, I will debug this part of the code to see what it happens

[pb82]

Thanks! Please let me know what you find. Also, you can always try the preferService: true CR option to force the Operator to make API calls through the Service instead of the Ingress.

[cmoulliard]

The preferService is defined to false and even if the ingress route is well declared the r.state.GrafanaRoute is nill

kc get ingress -n demo
NAME                     HOSTS                                  ADDRESS         PORTS   AGE
grafana-ingress          grafana-console.88.99.186.195.nip.io   88.99.186.195   80      2d10h
prometheus               prometheus.88.99.186.195.nip.io        88.99.186.195   80      2d16h
spring-boot-monitoring   sb-monitor.88.99.186.195.nip.io        88.99.186.195   80      2d16h

kc get grafana -n demo
NAME      AGE
grafana   2d10h

kc get grafana -n demo -o yaml | grep host
        {"apiVersion":"integreatly.org/v1alpha1","kind":"Grafana","metadata":{"annotations":{},"name":"grafana","namespace":"demo"},"spec":{"config":{"auth":{"disable_login_form":false,"disable_signout_menu":true},"auth.anonymous":{"enabled":true},"log":{"level":"warn","mode":"console"},"security":{"admin_password":"secret","admin_user":"root"}},"dashboardLabelSelector":[{"matchExpressions":[{"key":"app","operator":"In","values":["grafana"]}]}],"ingress":{"enabled":true,"hostname":"grafana-console.88.99.186.195.nip.io"}}}
      hostname: grafana-console.88.99.186.195.nip.io

[pb82]

@cmoulliard r.state.GrafanaRoute will always be nil on Kubernetes, Routes are only used when running on OpenShift.

On Kubernetes it should find the Ingress however: https://github.com/integr8ly/grafana-operator/blob/master/pkg/controller/grafana/grafana_controller.go#L218

Also, preferService needs to be set manually in the CR, see for example here: https://github.com/integr8ly/application-monitoring-operator/blob/master/templates/grafana.yaml#L64

[cmoulliard]

I added the parameter preferService: True and now the operator is using the correct IP address to access the dashboard but something wrong is happening during the post

Warning  ProcessingError  2m32s (x8 over 4m23s)  controller_grafanadashboard 
Post http://root:***@10.106.151.96:3000/api/dashboards/db: net/http: request canceled
while waiting for connection (Client.Timeout exceeded while awaiting headers)

The IP address is correct

kc get svc -n demo
NAME                     TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)    AGE
grafana-service          ClusterIP   10.106.151.96    <none>        3000/TCP   7m53s

Dashboard json = raw format of the req : https://gist.github.com/cmoulliard/bb37cf340376f1fa95a7218bb888ce7f

@pb82

[cmoulliard]

What I dont understand is that I can POST using the ingress address

curl -k -X POST -u root:xxxx \
>   -H "Content-Type: application/json" \
>   --data @raw.json \
>   http://grafana-console.88.99.186.195.nip.io/api/dashboards/db
{"id":1,"slug":"production-overview","status":"success","uid":"FIrd85wWz","url":"/d/FIrd85wWz/production-overview","version":1}%            

That also works If I do a curl POST from one of the pod within the namespace where grafana service has been deployed

[jboss@spring-boot-monitoring-787fff5797-xs2sn ~]$ curl -X POST -u root:xxx  \
    -H "Content-Type: application/json" \
    --data @raw.json \
     http://grafana-service:3000/api/dashboards/db
{"id":7,
"slug":"simple-dashboard-1",
"status":"success",
"uid":"TnL0l5QZk",
"url":"/d/TnL0l5QZk/simple-dashboard-1","version":1}
[jboss@spring-boot-monitoring-787fff5797-xs2sn ~]$ 

BUT

that will fail using the IP address as we got also a timeout

curl -X POST  -u root:secret  -H "Content-Type: application/json" --data @raw.json http://10.106.151.96:3000/api/dashboards/db
...

Question: How can grafana use the k8s service name (=> grafana-service) and not the IP address ?

[cmoulliard]

There is a problem with the logic or the way that you collect the ingress hostname. Why ?
When we access this part of the code and where the cluster is k8s + ingress then we got the cluster IP address and not at all the ingress hostname exposing the grafana service outside of the cluster.

So, the code will return this IP address by example 88.99.186.195 which is not enough when we use a cluster as the correct name for the ingress route should be : http://grafana-console.88.99.186.195.nip.io/ as declared within the grafana CR under .spec.ingress.hostname

[cmoulliard]

So, ideally, when the Grafana CR includes a spec.ingress.hostname, then the following part of the code should use it instead of the IP address. We must of course wait till the status of the ingress resource got an IP address

	if state.GrafanaIngress != nil && !preferService {
		for _, ingress:= range state.GrafanaIngress.Status.LoadBalancer.Ingress {
			if ingress.IP != "" {
				return fmt.Sprintf("http://%v", cr.Spec.Ingress.Hostname), nil
			}

@pb82 WDYT ?

[pb82]

@cmoulliard sorry for the late response. I think this is a good way forward, relying on the hostname in the CR instead of the one in the Ingress. As for preferService: this will only work on cluster. When you run the operator locally against a cluster it won't be able to access the service, hence the failed request. But if you use preferService and deploy the operator on the cluster, then this should always work.

Not sure if I get a chance to try it out this week, so if you want to give it a try please go ahead. I'll try to get to it as soon as possible.

[cmoulliard]

this will only work on cluster.

Correct. this is my fault as I was running the operator from my laptop

[cmoulliard]

Not sure if I get a chance to try it out this week, so if you want to give it a try please go ahead

I did a successful test using the following syntax for ingress

if state.GrafanaIngress != nil && !preferService {
		for _, ingress:= range state.GrafanaIngress.Status.LoadBalancer.Ingress {
			if ingress.IP != "" {
				return fmt.Sprintf("http://%v", cr.Spec.Ingress.Hostname), nil
			}

[pb82]

@cmoulliard @1337andre The suggested fix is included in the v3.1.0 release (thanks @cmoulliard for proposing and testing this)! The only assumption I make is that the ingress hostname will always be https. Do you think we should add an insecure / http option too?

Also, would you mind trying of v3.1.0 resolves your issue?

[cmoulliard]

The only assumption I make is that the ingress hostname will always be https. Do you think we should add an insecure / http option too?

ingress can take care of that (I mean to send packets to http or https) https://kubernetes.github.io/ingress-nginx/user-guide/tls/#server-side-https-enforcement-through-redirect. So, I dont think that we must add a second option but instead document and log it correctly

[afoninsky]

I have http ingress (tls terminates at top-level balancer) and have the same error in v3.1.0

apiVersion: integreatly.org/v1alpha1
kind: GrafanaDashboard
metadata:
  name: dashboard-with-plugins
  labels:
    app: grafana
spec:
  name: test
  json: "{}"
  plugins:
    - name: "agenty-flowcharting-panel"
      version: "0.6.1"

apiVersion: integreatly.org/v1alpha1
kind: Grafana
...
spec:
  # client:
  #   preferService: true

  ingress:
    enabled: true
    hostname: ...

  config:
    server:
      root_url: ...
      enable_gzip: True

[pb82]

@afoninsky Ok, so we probably should not assume always https. As a workaround, I think you can set preferService: https://github.com/integr8ly/grafana-operator/blob/master/documentation/deploy_grafana.md#configuring-grafana-api-access (this forces the operator to use the service instead of the ingress).

[afoninsky]

Sorry didn't mention, I have another error using "preferService": error creating dashboard, expected status 200 but got 400

[pb82]

Can you share the dashboard?

[cmoulliard]

The message reported by the pod log is not enough and ideally the service, ingress URL should be logged to let the user to understand why controller cannot access the Grafan Admin UI/

[afoninsky]

Can you share the dashboard?

Shared it higher: #136 (comment)
Perhaps it's something related to the way how operator stores permanent data: I use postgresql

[hubeadmin]

@cmoulliard @afoninsky Hi guys, bringing this topic back around, have you seen this behaviour with recent grafana versions?

[1337andre]

no problems with 3.6

LGTM

[david-martin]

Closing as resolved