Alerts: Exclude '/schedulerpb.SchedulerForQuerier/QuerierLoop' route in LokiRequestLatency alert #8138
Conversation
…uestLatency alert
|
|
|
./tools/diff_coverage.sh ../loki-target-branch/test_results.txt test_results.txt ingester,distributor,querier,querier/queryrange,iter,storage,chunkenc,logql,loki Change in test coverage per package. Green indicates 0 or positive change, red indicates that test coverage for a package fell. + ingester 0%
+ distributor 0%
+ querier 0%
+ querier/queryrange 0%
+ iter 0%
+ storage 0%
+ chunkenc 0%
+ logql 0%
+ loki 0% |
|
Thanks for your contribution, this looks good! Can you please run |
Done! |
|
./tools/diff_coverage.sh ../loki-target-branch/test_results.txt test_results.txt ingester,distributor,querier,querier/queryrange,iter,storage,chunkenc,logql,loki Change in test coverage per package. Green indicates 0 or positive change, red indicates that test coverage for a package fell. + ingester 0%
+ distributor 0%
+ querier 0%
+ querier/queryrange 0%
+ iter 0%
+ storage 0%
+ chunkenc 0%
+ logql 0%
+ loki 0% |
…in LokiRequestLatency alert (grafana#8138) Excludes the route `/schedulerpb.SchedulerForQuerier/QuerierLoop` from the `LokiRequestLatency` alert query config. We have found that this route often has extremely high request latencies which will trigger the alert constantly for us. Looking through the source code it seems like it is expected to have high request latencies for this route and so in our configuration have excluded the route. We can see that this route is also being excluded in the [respective mimir config](https://github.com/grafana/mimir/blob/main/operations/mimir-mixin-compiled/alerts.yaml#L33) so believe it might be relevant to exclude here in loki as well
What this PR does / why we need it:
Excludes the route
/schedulerpb.SchedulerForQuerier/QuerierLoopfrom theLokiRequestLatencyalert query config. We have found that this route often has extremely high request latencies which will trigger the alert constantly for us. Looking through the source code it seems like it is expected to have high request latencies for this route and so in our configuration have excluded the route. We can see that this route is also being excluded in the respective mimir config so believe it might be relevant to exclude here in loki as wellWhich issue(s) this PR fixes:
N/A
Special notes for your reviewer:
Checklist
CONTRIBUTING.mdguide (required)CHANGELOG.mdupdateddocs/sources/upgrading/_index.md