Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Export webhook metrics to prometheus #4707

Merged
merged 1 commit into from
Jul 12, 2019
Merged

Export webhook metrics to prometheus #4707

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
54 changes: 30 additions & 24 deletions cmd/webhook/main.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -35,6 +35,7 @@ import (
"knative.dev/pkg/configmap"
"knative.dev/pkg/logging"
"knative.dev/pkg/logging/logkey"
"knative.dev/pkg/metrics"
"knative.dev/pkg/signals"
"knative.dev/pkg/system"
"knative.dev/pkg/version"
Expand Down Expand Up @@ -85,6 +86,9 @@ func main() {

// Watch the logging config map and dynamically update logging levels.
configMapWatcher := configmap.NewInformedWatcher(kubeClient, system.Namespace())
// Watch the observability config map and dynamically update metrics exporter.
configMapWatcher.Watch(metrics.ConfigMapName(), metrics.UpdateExporterFromConfigMap(component, logger))
// Watch the observability config map and dynamically update request logs.
configMapWatcher.Watch(logging.ConfigMapName(), logging.UpdateLevelFromConfigMap(logger, atomicLevel, component))

store := apiconfig.NewStore(logger.Named("config-store"))
Expand All @@ -102,31 +106,33 @@ func main() {
SecretName: "webhook-certs",
WebhookName: "webhook.serving.knative.dev",
}
controller := webhook.AdmissionController{
Client: kubeClient,
Options: options,
Handlers: map[schema.GroupVersionKind]webhook.GenericCRD{
v1alpha1.SchemeGroupVersion.WithKind("Revision"): &v1alpha1.Revision{},
v1alpha1.SchemeGroupVersion.WithKind("Configuration"): &v1alpha1.Configuration{},
v1alpha1.SchemeGroupVersion.WithKind("Route"): &v1alpha1.Route{},
v1alpha1.SchemeGroupVersion.WithKind("Service"): &v1alpha1.Service{},
v1beta1.SchemeGroupVersion.WithKind("Revision"): &v1beta1.Revision{},
v1beta1.SchemeGroupVersion.WithKind("Configuration"): &v1beta1.Configuration{},
v1beta1.SchemeGroupVersion.WithKind("Route"): &v1beta1.Route{},
v1beta1.SchemeGroupVersion.WithKind("Service"): &v1beta1.Service{},
autoscalingv1alpha1.SchemeGroupVersion.WithKind("PodAutoscaler"): &autoscalingv1alpha1.PodAutoscaler{},
net.SchemeGroupVersion.WithKind("Certificate"): &net.Certificate{},
net.SchemeGroupVersion.WithKind("ClusterIngress"): &net.ClusterIngress{},
net.SchemeGroupVersion.WithKind("ServerlessService"): &net.ServerlessService{},
},
Logger: logger,
DisallowUnknownFields: true,

// Decorate contexts with the current state of the config.
WithContext: func(ctx context.Context) context.Context {
return v1beta1.WithUpgradeViaDefaulting(store.ToContext(ctx))
},

handlers := map[schema.GroupVersionKind]webhook.GenericCRD{
v1alpha1.SchemeGroupVersion.WithKind("Revision"): &v1alpha1.Revision{},
v1alpha1.SchemeGroupVersion.WithKind("Configuration"): &v1alpha1.Configuration{},
v1alpha1.SchemeGroupVersion.WithKind("Route"): &v1alpha1.Route{},
v1alpha1.SchemeGroupVersion.WithKind("Service"): &v1alpha1.Service{},
v1beta1.SchemeGroupVersion.WithKind("Revision"): &v1beta1.Revision{},
v1beta1.SchemeGroupVersion.WithKind("Configuration"): &v1beta1.Configuration{},
v1beta1.SchemeGroupVersion.WithKind("Route"): &v1beta1.Route{},
v1beta1.SchemeGroupVersion.WithKind("Service"): &v1beta1.Service{},
autoscalingv1alpha1.SchemeGroupVersion.WithKind("PodAutoscaler"): &autoscalingv1alpha1.PodAutoscaler{},
net.SchemeGroupVersion.WithKind("Certificate"): &net.Certificate{},
net.SchemeGroupVersion.WithKind("ClusterIngress"): &net.ClusterIngress{},
net.SchemeGroupVersion.WithKind("ServerlessService"): &net.ServerlessService{},
}

// Decorate contexts with the current state of the config.
ctxFunc := func(ctx context.Context) context.Context {
return v1beta1.WithUpgradeViaDefaulting(store.ToContext(ctx))
}

controller, err := webhook.NewAdmissionController(kubeClient, options, handlers, logger, ctxFunc, true)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's unfortunate that "DisallowUnknownFields" gets collapsed to true in this constructor. Not for this PR, but it would be nice to have a signature like:

controller, err := webhook.AdmissionController(
	kubeClient,
	handlers,
	logger,
	webhook.EnrichContext(ctxFunc),
	webhook.Namespace(system.Namespace()),
	webhook.Secret("webhook-certs"),
	webhook.WebhookName("webhook.serving.knative.dev"),
	webhook.UnknownFields)

Where the default context function, port, and some other items were already set, and we only needed to override some fields. Since you're now returning an error, you could also complain at this point if Secret, Name, etc were not set.


if err != nil {
logger.Fatalw("Failed to create admission controller", zap.Error(err))
}

if err = controller.Run(stopCh); err != nil {
logger.Fatalw("Failed to start the admission controller", zap.Error(err))
}
Expand Down
18 changes: 18 additions & 0 deletions config/monitoring/metrics/prometheus/100-prometheus-scrape-config.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -81,6 +81,24 @@ data:
target_label: pod
- source_labels: [__meta_kubernetes_service_name]
target_label: service
# Webhook pods
- job_name: webhook
scrape_interval: 3s
scrape_timeout: 3s
kubernetes_sd_configs:
- role: pod
relabel_configs:
# Scrape only the the targets matching the following metadata
- source_labels: [__meta_kubernetes_namespace, __meta_kubernetes_pod_label_role, __meta_kubernetes_pod_container_port_name]
action: keep
regex: knative-serving;webhook;metrics-port
# Rename metadata labels to be reader friendly
- source_labels: [__meta_kubernetes_namespace]
target_label: namespace
- source_labels: [__meta_kubernetes_pod_name]
target_label: pod
- source_labels: [__meta_kubernetes_service_name]
target_label: service
# Queue proxy metrics
- job_name: queue-proxy
scrape_interval: 3s
Expand Down
7 changes: 7 additions & 0 deletions config/webhook.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -41,6 +41,9 @@ spec:
# This is the Go import path for the binary that is containerized
# and substituted here.
image: github.com/knative/serving/cmd/webhook
ports:
- name: metrics-port
containerPort: 9090
resources:
# Request 2x what we saw running e2e
requests:
Expand All @@ -60,6 +63,10 @@ spec:
fieldPath: metadata.namespace
- name: CONFIG_LOGGING_NAME
value: config-logging
- name: CONFIG_OBSERVABILITY_NAME
value: config-observability
- name: METRICS_DOMAIN
value: knative.dev/serving
securityContext:
allowPrivilegeEscalation: false
volumes:
Expand Down