TestFleetMode is failing Fleet setup

[pebrc]

No Pod in e2e-venus

We do not even get to deploying the pod as the Fleet setup call is timing out.

Events from e2e-venus (Fleet server)

"AssociationStatusChange  Association status changed from [] to [Established] 1 2025-02-11T23:05:32Z 2025-02-11T23:05:32Z"
"Delayed  ECK cannot setup Fleet enrollment. Waiting for Kibana credentials. This should be a transient issue. 13 2025-02-11T23:05:34Z 2025-02-11T23:05:57Z"
"ReconciliationError  Reconciliation error: Post \"https://test-agent-fleet-zd9t-kb-http.e2e-iiz43-mercury.svc:5601/api/fleet/setup\": net/http: request canceled (Client.Timeout exceeded while awaiting headers) 4 2025-02-11T23:07:58Z 2025-02-11T23:20:32Z"

We are trying for ~15 minute to set up Fleet in Kibana

Kibana pod status

 "name": "kibana",
        "ready": true,
        "restartCount": 0,
        "started": true,
        "state": {
          "running": {
            "startedAt": "2025-02-11T23:04:54Z"
          }
        }

Kibana looks good on the surface.

Kibana logs:

[2025-02-11T23:05:57.158+00:00][ERROR][plugins.security.authentication] Authentication attempt failed: {"error":{"root_cause":[{"type":"security_exception","reason":"unable to authenticate user [e2e-iiz43-venus-test-agent-fleet-fs-rhcd-agent-kb-user] for REST request [/_security/_authenticate]","header":{"WWW-Authenticate":["Basic realm=\"security\", charset=\"UTF-8\"","Bearer realm=\"security\"","ApiKey"]}}],"type":"security_exception","reason":"unable to authenticate user [e2e-iiz43-venus-test-agent-fleet-fs-rhcd-agent-kb-user] for REST request [/_security/_authenticate]","header":{"WWW-Authenticate":["Basic realm=\"security\", charset=\"UTF-8\"","Bearer realm=\"security\"","ApiKey"]}},"status":401}

==== Last Fleet event mentioning the transient auth issue ====
[2025-02-11T23:06:17.687+00:00][ERROR][plugins.security.authentication] Authentication attempt failed: {"error":{"root_cause":[{"type":"security_exception","reason":"unable to authenticate user [e2e-iiz43-venus-test-agent-fleet-fs-rhcd-agent-kb-user] for REST request [/_security/_authenticate]","header":{"WWW-Authenticate":["Basic realm=\"security\", charset=\"UTF-8\"","Bearer realm=\"security\"","ApiKey"]}}],"type":"security_exception","reason":"unable to authenticate user [e2e-iiz43-venus-test-agent-fleet-fs-rhcd-agent-kb-user] for REST request [/_security/_authenticate]","header":{"WWW-Authenticate":["Basic realm=\"security\", charset=\"UTF-8\"","Bearer realm=\"security\"","ApiKey"]}},"status":401}
[2025-02-11T23:10:28.262+00:00][INFO ][plugins.securitySolution] Fetch risk engine metrics
[2025-02-11T23:10:28.504+00:00][INFO ][plugins.cloudSecurityPosture] Cloud Security telemetry: Accounts payload was sent successfully
[2025-02-11T23:10:28.505+00:00][INFO ][plugins.cloudSecurityPosture] Cloud Security telemetry: Resources payload was sent successfully
[2025-02-11T23:10:28.507+00:00][INFO ][plugins.cloudSecurityPosture] Cloud Security telemetry: Rules payload was sent successfully
[2025-02-11T23:10:28.509+00:00][INFO ][plugins.cloudSecurityPosture] Cloud Security telemetry: Cloud Accounts payload was sent successfully
[2025-02-11T23:10:28.861+00:00][INFO ][plugins.cloudSecurityPosture] Cloud Security telemetry: Alerts payload was sent successfully
[2025-02-11T23:10:28.992+00:00][INFO ][plugins.cloudSecurityPosture] Cloud Security telemetry: Installation payload was sent successfully
[2025-02-11T23:10:29.097+00:00][INFO ][plugins.cloudSecurityPosture] Cloud Security telemetry: Muted Rules payload was sent successfully
[2025-02-11T23:10:29.114+00:00][INFO ][plugins.cloudSecurityPosture] Cloud Security telemetry: Indices payload was sent successfully
[2025-02-11T23:15:32.510+00:00][INFO ][plugins.fleet.fleet:unenroll-inactive-agents-task:1.0.0] [runTask()] started
[2025-02-11T23:15:32.545+00:00][INFO ][plugins.fleet.fleet:unenroll-inactive-agents-task:1.0.0] [UnenrollInactiveAgentsTask] runTask ended: success
[2025-02-11T23:20:34.968+00:00][INFO ][plugins.fleet] Fleet Usage: {"agents_enabled":true,"agents":{"total_enrolled":0,"healthy":0,"unhealthy":0,"offline":0,"inactive":0,"unenrolled":0,"total_all_statuses":0,"updating":0},"fleet_server":{"total_all_statuses":0,"total_enrolled":0,"healthy":0,"unhealthy":0,"offline":0,"updating":0,"inactive":0,"unenrolled":0,"num_host_urls":0},"license_issued_to":"test-agent-fleet-5pjs"}
==== END logs for e2e-iiz43-mercury/test-agent-fleet-zd9t-kb-779bbf4b48-bt568 ====

Can't see anything of note here either.

Operator

23:06:58.677 | info | [agent-controller]:null | Starting reconciliation run | {"trace.id":"a358d10157957214d272d73f6d97b3ea","transaction.id":"a358d10157957214","iteration":"372","namespace":"e2e-iiz43-venus","agent_name":"test-agent-fleet-fs-rhcd"}
23:06:58.678 | debug | [agent-v1alpha1-validation]:null | Validate create | {"name":"test-agent-fleet-fs-rhcd"}
23:06:58.679 | debug | [agent-controller]:null | Fleet API HTTP request | {"trace.id":"a358d10157957214d272d73f6d97b3ea","transaction.id":"a358d10157957214","iteration":"372","namespace":"e2e-iiz43-venus","agent_name":"test-agent-fleet-fs-rhcd","span.id":"c3bbee75b1f9dfe7","method":"POST","url":"https://test-agent-fleet-zd9t-kb-http.e2e-iiz43-mercury.svc:5601/api/fleet/setup"}
23:07:58.680 | info | [agent-controller]:null | Ending reconciliation run | {"trace.id":"a358d10157957214d272d73f6d97b3ea","transaction.id":"a358d10157957214","iteration":"372","namespace":"e2e-iiz43-venus","agent_name":"test-agent-fleet-fs-rhcd","took":60.002722081}
23:07:58.680 | error | [manager.eck-operator]:null | Reconciler error | {"controller":"agent-controller","object":{"name":"test-agent-fleet-fs-rhcd","namespace":"e2e-iiz43-venus"},"namespace":"e2e-iiz43-venus","name":"test-agent-fleet-fs-rhcd","reconcileID":"afa7dace-893c-4038-a241-d9364a8ef6d7","error":"Post \"https://test-agent-fleet-zd9t-kb-http.e2e-iiz43-mercury.svc:5601/api/fleet/setup\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)","errorCauses":[{"error":"Post \"https://test-agent-fleet-zd9t-kb-http.e2e-iiz43-mercury.svc:5601/api/fleet/setup\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"}],"error.stack_trace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/root/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.20.1/pkg/internal/controller/controller.go:341\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/root/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.20.1/pkg/internal/controller/controller.go:288\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2\n\t/root/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.20.1/pkg/internal/controller/controller.go:249"}

23:20:32.131 | error | [manager.eck-operator]:null | Reconciler error | {"controller":"agent-controller","object":{"name":"test-agent-fleet-fs-rhcd","namespace":"e2e-iiz43-venus"},"namespace":"e2e-iiz43-venus","name":"test-agent-fleet-fs-rhcd","reconcileID":"9474ff0a-564d-4a83-8f6a-172df582bf5c","error":"Post \"https://test-agent-fleet-zd9t-kb-http.e2e-iiz43-mercury.svc:5601/api/fleet/setup\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)","errorCauses":[{"error":"Post \"https://test-agent-fleet-zd9t-kb-http.e2e-iiz43-mercury.svc:5601/api/fleet/setup\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"}],"error.stack_trace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/root/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.20.1/pkg/internal/controller/controller.go:341\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/root/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.20.1/pkg/internal/controller/controller.go:288\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2\n\t/root/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.20.1/pkg/internal/controller/controller.go:249"}