feat/fix(datasource): support for hardcoded uid, multiple fixes

[weisdd]

• Datasources:
  • features:
    • Add support for hardcoded uid;
    • Add "Last resync" to kubectl outputs;
    • Take over uid for manually created datasources;
  • fixes:
    • Calculate hash after all referenced secrets are resolved;
    • Fetch referenced secrets once per datasource (previously - per instance);
    • In onDatasourceDeleted, tolerate case where a datasource was manually deleted (not able to find ID);
    • Do periodic resync when there are no changes in datasource spec;
    • Update CR status once per CR instance, not per Grafana instance (at Reconcile level instead of onDatasourceCreated) to improve overall reconciliation flow;
  • chores:
    • improve namings in a few functions.

Fixes: #1056

[galvesLoy]

Hello @weisdd

After testing different datasources, I got an issue that was not existing on v5.0.0-rc1 for elasticsearch one

1.6853557627679813e+09  INFO    GrafanaDatasourceReconciler     found matching Grafana instances for datasource {"controller": "grafanadatasource", "controllerGroup": "grafana.integreatly.org", "controllerKind": "GrafanaDatasource", "GrafanaDatasource": {"name":"grafana-datasource-elasticsearch-infra-ec","namespace":"monitoring"}, "namespace": "monitoring", "name": "grafana-datasource-elasticsearch-infra-ec", "reconcileID": "170b38f2-d9eb-41de-8613-3abe2399144b", "count": 1}
1.6853557629693453e+09  ERROR   GrafanaDatasourceReconciler     error reconciling datasource    {"controller": "grafanadatasource", "controllerGroup": "grafana.integreatly.org", "controllerKind": "GrafanaDatasource", "GrafanaDatasource": {"name":"grafana-datasource-elasticsearch-infra-ec","namespace":"monitoring"}, "namespace": "monitoring", "name": "grafana-datasource-elasticsearch-infra-ec", "reconcileID": "170b38f2-d9eb-41de-8613-3abe2399144b", "datasource": "grafana-datasource-elasticsearch-infra-ec", "grafana": "grafana-application", "error": "status: 400, body: {\"message\":\"Validation error, invalid URL\",\"traceID\":\"\"}"}
github.com/grafana-operator/grafana-operator/v5/controllers.(*GrafanaDatasourceReconciler).Reconcile
        github.com/grafana-operator/grafana-operator/v5/controllers/datasource_controller.go:245
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
        sigs.k8s.io/controller-runtime@v0.13.0/pkg/internal/controller/controller.go:121
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
        sigs.k8s.io/controller-runtime@v0.13.0/pkg/internal/controller/controller.go:320
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
        sigs.k8s.io/controller-runtime@v0.13.0/pkg/internal/controller/controller.go:273
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
        sigs.k8s.io/controller-runtime@v0.13.0/pkg/internal/controller/controller.go:234

And here is the configuration

apiVersion: grafana.integreatly.org/v1beta1
kind: GrafanaDatasource
metadata:
  name: grafana-datasource-elasticsearch-infra-ec
  namespace: monitoring
spec:
  secrets:
    - grafana-application-elasticsearch-infra-ec
  datasource:
    name: elasticsearch-infra-ec
    type: elasticsearch
    uid: MHme6a9Qg   # Fixed for every Grafana
    access: proxy
    url: ${ec_url}   # URL is working properly and all access are well configured
    basicAuth: true
    basicAuthUser: ${ec_user}
    database: '*'
    secureJsonData:
      basicAuthPassword: ${ec_password}
    jsonData:
      interval: Daily
      timeField: '@timestamp'
      esVersion: 8.0.0
      logMessageField: message
      logLevelField: log.level
      includeFrozen: true
      maxConcurrentShardRequests: 5
      xpack: true
      dataLinks:
        - datasourceUid: 3xqX2UPGk #Actually the Tempo UID that is well configured 
          field: trace.id
          url: ${__value.raw}  # <-- might be the issue ?
  instanceSelector:
    matchLabels:
      app.kubernetes.io/instance: grafana-application
      app.kubernetes.io/name: grafana

I think this may come from the "Data Links" (I show you in image what are the differencies) related to internal link activated or not)

Note 0 : the GrafanaDatasource you see does not send 400 errors on v5.0.0-rc1 and reconcile without issue

Note 1 : I tested to created manually the elasticsearch datasource using the same testing and it works well

Note 2: Other datasources are working properly on my side :) (AWS, Azure, Prometheus, Alertmanager, etc), good job 🐐

Edit: This was due to secret value definition (secret to valuesFrom, nothing related to the PR)

[weisdd]

@galvesLoy thanks for the test!
I think it has to do with this change: #1055 (released as part of v5.0.0-rc3), so you need to change your manifest according to this example.

[galvesLoy]

@weisdd It seems to work :) thanks

[hubeadmin]

LGTM, tested locally, works as expected