[WIP] Attempt to reduce 502s in queue-proxy due to connect errors

[tcnghia]

Proposed Changes

• Also retry on 'connection refused' errors, instead of just 'connect timeout'.
• Reset the IdleConnectionTimeout back to 90s, since Replace a prober factory with a prober with no caching. #4756 makes the change in Reduce the idle timeouts in the default transports. #3996 unnecessary.

Release Note

NONE

[knative-prow-robot]

@tcnghia: 0 warnings.

In response to this:

Proposed Changes

• Also retry on 'connection refused' errors, instead of just 'connect timeout'.

Release Note

NONE

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[tcnghia]

/test pull-knative-serving-istio-1.1-mesh

[tcnghia]

/test pull-knative-serving-istio-1.2-mesh

[tcnghia]

/test pull-knative-serving-istio-1.1-mesh
/test pull-knative-serving-istio-1.2-mesh

[vagababov]

/lgtm
/hold

[markusthoemmes]

/lgtm
/approve

[knative-prow-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: markusthoemmes, tcnghia

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• pkg/network/OWNERS [markusthoemmes,tcnghia]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[tcnghia]

Still hitting 502s 4 times

kubelogs.go:158: I 06:30:43.513 [kpa-class-podautoscaler-controller] [serving-tests/autoscale-sustaining-qgicvgkz-c852f] Reconcile succeeded. Time taken: 473.352µs.
autoscale_test.go:140: Status = 502, want: 200
autoscale_test.go:141: Response: status: 502, body: dial tcp 127.0.0.1:8080: connect: connection refused
    , headers: map[Content-Length:[53] Content-Type:[text/plain; charset=utf-8] Date:[Thu, 12 Sep 2019 06:30:43 GMT] Server:[istio-envoy] X-Content-Type-Options:[nosniff] X-Envoy-Upstream-Service-Time:[4] Zipkin_trace_id:[dbc7d1e6eb91c83f7906320e03104e33]]
spoof.go:122: Spoofing autoscale-sustaining-qgicvgkz.serving-tests.35.188.4.244.xip.io:80 -> 35.188.4.244:80
spoof.go:122: Spoofing autoscale-sustaining-qgicvgkz.serving-tests.35.188.4.244.xip.io:80 -> 35.188.4.244:80
spoof.go:122: Spoofing autoscale-sustaining-qgicvgkz.serving-tests.35.188.4.244.xip.io:80 -> 35.188.4.244:80
autoscale_test.go:140: Status = 502, want: 200
autoscale_test.go:141: Response: status: 502, body: dial tcp 127.0.0.1:8080: connect: connection refused
    , headers: map[Content-Length:[53] Content-Type:[text/plain; charset=utf-8] Date:[Thu, 12 Sep 2019 06:30:43 GMT] Server:[istio-envoy] X-Content-Type-Options:[nosniff] X-Envoy-Upstream-Service-Time:[2] Zipkin_trace_id:[70239972f43721741312dec5f07b70aa]]
autoscale_test.go:140: Status = 502, want: 200
autoscale_test.go:141: Response: status: 502, body: dial tcp 127.0.0.1:8080: connect: connection refused
    , headers: map[Content-Length:[53] Content-Type:[text/plain; charset=utf-8] Date:[Thu, 12 Sep 2019 06:30:43 GMT] Server:[istio-envoy] X-Content-Type-Options:[nosniff] X-Envoy-Upstream-Service-Time:[3] Zipkin_trace_id:[dcf7441c5bbcb44b2f8fb3bb29698646]]
autoscale_test.go:140: Status = 502, want: 200
autoscale_test.go:141: Response: status: 502, body: dial tcp 127.0.0.1:8080: connect: connection refused
    , headers: map[Content-Length:[53] Content-Type:[text/plain; charset=utf-8] Date:[Thu, 12 Sep 2019 06:30:43 GMT] Server:[istio-envoy] X-Content-Type-Options:[nosniff] X-Envoy-Upstream-Service-Time:[2] Zipkin_trace_id:[2923b60162cd7c9aab5aa3a91f2e1482]]
spoof.go:122: Spoofing autoscale-sustaining-qgicvgkz.serving-tests.35.188.4.244.xip.io:80 -> 35.188.4.244:80

[knative-prow-robot]

New changes are detected. LGTM label has been removed.

[tcnghia]

Looks like either net.Error#Temporary() doesn't include connection refused or the retries didn't help.

[tcnghia]

/test pull-knative-serving-istio-1.1-mesh
/test pull-knative-serving-istio-1.2-mesh

[knative-metrics-robot]

The following is the coverage report on pkg/.
Say /test pull-knative-serving-go-coverage to re-run this coverage report

File	Old Coverage	New Coverage	Delta
pkg/network/transports.go	96.0%	95.7%	-0.3

[knative-prow-robot]

@tcnghia: The following tests failed, say /retest to rerun them all:

Test name	Commit	Details	Rerun command
pull-knative-serving-istio-1.1-mesh	2492232	link	/test pull-knative-serving-istio-1.1-mesh
pull-knative-serving-istio-1.2-mesh	2492232	link	/test pull-knative-serving-istio-1.2-mesh

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

[tcnghia]

Looks like retrying won't fix

    autoscale_test.go:141: Response: status: 502, body: timed out dialing```

[tcnghia]

/close

[knative-prow-robot]

@tcnghia: Closed this PR.

In response to this:

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.