Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

FaultIntegrationTestAllProtocols / Timed out waiting for listeners flake #12480

Closed
mattklein123 opened this issue Aug 4, 2020 · 4 comments · Fixed by #12527
Closed

FaultIntegrationTestAllProtocols / Timed out waiting for listeners flake #12480

mattklein123 opened this issue Aug 4, 2020 · 4 comments · Fixed by #12527
Assignees
@mattklein123
Labels
area/test flakes help wanted Needs help!

Comments

@mattklein123
Copy link
Member 

[ RUN      ] Protocols/FaultIntegrationTestAllProtocols.HeaderFaultAbortGrpcConfig/IPv4_HttpDownstream_Http2Upstream
TestRandomGenerator running with seed 901267552
[2020-08-04 22:52:52.748][22248][critical][assert] [test/integration/integration.cc:500] assert failure: 0. Details: Timed out waiting for listeners.
[2020-08-04 22:52:52.748][22248][critical][backtrace] [bazel-out/aarch64-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:104] Caught Aborted, suspect faulting address 0x3e9000056e8
[2020-08-04 22:52:52.748][22248][critical][backtrace] [bazel-out/aarch64-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:91] Backtrace (use tools/stack_decode.py to get line numbers):
[2020-08-04 22:52:52.748][22248][critical][backtrace] [bazel-out/aarch64-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:92] Envoy version: 0/1.16.0-dev/test/RELEASE/BoringSSL
[2020-08-04 22:52:52.748][22248][critical][backtrace] [bazel-out/aarch64-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #0: __kernel_rt_sigreturn [0xffffa90775b8]
[2020-08-04 22:52:52.748][22248][critical][backtrace] [bazel-out/aarch64-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #1: abort [0xffffa8e14884]
[2020-08-04 22:52:52.753][22248][critical][backtrace] [bazel-out/aarch64-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #2: Envoy::BaseIntegrationTest::createGeneratedApiTestServer() [0x8b292c]
[2020-08-04 22:52:52.758][22248][critical][backtrace] [bazel-out/aarch64-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #3: Envoy::BaseIntegrationTest::createEnvoy() [0x8b1c3c]
[2020-08-04 22:52:52.763][22248][critical][backtrace] [bazel-out/aarch64-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #4: Envoy::Extensions::HttpFilters::Fault::(anonymous namespace)::FaultIntegrationTestAllProtocols_HeaderFaultAbortGrpcConfig_Test::TestBody() [0x84eecc]
[2020-08-04 22:52:52.767][22248][critical][backtrace] [bazel-out/aarch64-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #5: testing::internal::HandleExceptionsInMethodIfSupported<>() [0x15e0df4]
[2020-08-04 22:52:52.772][22248][critical][backtrace] [bazel-out/aarch64-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #6: testing::Test::Run() [0x15e0d0c]
[2020-08-04 22:52:52.777][22248][critical][backtrace] [bazel-out/aarch64-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #7: testing::TestInfo::Run() [0x15e1c4c]
[2020-08-04 22:52:52.782][22248][critical][backtrace] [bazel-out/aarch64-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #8: testing::TestSuite::Run() [0x15e2588]
[2020-08-04 22:52:52.786][22248][critical][backtrace] [bazel-out/aarch64-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #9: testing::internal::UnitTestImpl::RunAllTests() [0x15ef004]
[2020-08-04 22:52:52.791][22248][critical][backtrace] [bazel-out/aarch64-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #10: testing::internal::HandleExceptionsInMethodIfSupported<>() [0x15ee8a0]
[2020-08-04 22:52:52.796][22248][critical][backtrace] [bazel-out/aarch64-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #11: testing::UnitTest::Run() [0x15ee708]
[2020-08-04 22:52:52.801][22248][critical][backtrace] [bazel-out/aarch64-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #12: Envoy::TestRunner::RunTests() [0xe53ddc]
[2020-08-04 22:52:52.805][22248][critical][backtrace] [bazel-out/aarch64-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #13: main [0xe53268]
[2020-08-04 22:52:52.805][22248][critical][backtrace] [bazel-out/aarch64-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #14: __libc_start_main [0xffffa8e016e0]
================================================================================
@mattklein123
Copy link
Member Author

Saw this on an ARM run:

[ RUN      ] IpVersions/HttpTimeoutIntegrationTest.DeriveTimeoutInIngressEnvoy/IPv6
TestRandomGenerator running with seed 1218666088
[2020-08-06 00:53:54.119][17193][critical][assert] [test/integration/integration.cc:500] assert failure: 0. Details: Timed out waiting for listeners.
[2020-08-06 00:53:54.119][17193][critical][backtrace] [bazel-out/aarch64-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:104] Caught Aborted, suspect faulting address 0x3e900004329
[2020-08-06 00:53:54.119][17193][critical][backtrace] [bazel-out/aarch64-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:91] Backtrace (use tools/stack_decode.py to get line numbers):
[2020-08-06 00:53:54.119][17193][critical][backtrace] [bazel-out/aarch64-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:92] Envoy version: 0/1.16.0-dev/test/RELEASE/BoringSSL
[2020-08-06 00:53:54.119][17193][critical][backtrace] [bazel-out/aarch64-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #0: __kernel_rt_sigreturn [0xffff8a2435b8]
[2020-08-06 00:53:54.119][17193][critical][backtrace] [bazel-out/aarch64-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #1: abort [0xffff89fe0884]
[2020-08-06 00:53:54.125][17193][critical][backtrace] [bazel-out/aarch64-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #2: Envoy::BaseIntegrationTest::createGeneratedApiTestServer() [0x879dbc]
[2020-08-06 00:53:54.142][17193][critical][backtrace] [bazel-out/aarch64-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #3: Envoy::BaseIntegrationTest::createEnvoy() [0x8790cc]
[2020-08-06 00:53:54.148][17193][critical][backtrace] [bazel-out/aarch64-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #4: Envoy::HttpTimeoutIntegrationTest::initialize() [0x809c24]
[2020-08-06 00:53:54.153][17193][critical][backtrace] [bazel-out/aarch64-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #5: Envoy::HttpTimeoutIntegrationTest_DeriveTimeoutInIngressEnvoy_Test::TestBody() [0x8001f0]
[2020-08-06 00:53:54.174][17193][critical][backtrace] [bazel-out/aarch64-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #6: testing::internal::HandleExceptionsInMethodIfSupported<>() [0x15b1964]
[2020-08-06 00:53:54.180][17193][critical][backtrace] [bazel-out/aarch64-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #7: testing::Test::Run() [0x15b187c]
[2020-08-06 00:53:54.185][17193][critical][backtrace] [bazel-out/aarch64-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #8: testing::TestInfo::Run() [0x15b27bc]
[2020-08-06 00:53:54.202][17193][critical][backtrace] [bazel-out/aarch64-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #9: testing::TestSuite::Run() [0x15b30f8]
[2020-08-06 00:53:54.208][17193][critical][backtrace] [bazel-out/aarch64-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #10: testing::internal::UnitTestImpl::RunAllTests() [0x15bfb74]
[2020-08-06 00:53:54.213][17193][critical][backtrace] [bazel-out/aarch64-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #11: testing::internal::HandleExceptionsInMethodIfSupported<>() [0x15bf410]
[2020-08-06 00:53:54.247][17193][critical][backtrace] [bazel-out/aarch64-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #12: testing::UnitTest::Run() [0x15bf278]
[2020-08-06 00:53:54.252][17193][critical][backtrace] [bazel-out/aarch64-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #13: Envoy::TestRunner::RunTests() [0xe24610]
[2020-08-06 00:53:54.257][17193][critical][backtrace] [bazel-out/aarch64-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #14: main [0xe23a9c]
[2020-08-06 00:53:54.257][17193][critical][backtrace] [bazel-out/aarch64-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #15: __libc_start_main [0xffff89fcd6e0]

There must be same race on listener startup.

@mattklein123 mattklein123 changed the title FaultIntegrationTestAllProtocols flake FaultIntegrationTestAllProtocols / Timed out waiting for listeners flake Aug 6, 2020
@mattklein123
Copy link
Member Author

I think this is an issue w/ simulated time and waiting for listeners to come up. We probably need to actually do a real sleep while waiting for listeners to come up. cc @jmarantz

@mattklein123
Copy link
Member Author

Discussed with @jmarantz offline. I will work on some type of fix here.

@mattklein123 mattklein123 self-assigned this Aug 6, 2020
@mattklein123 mattklein123 mentioned this issue Aug 6, 2020
Closed
@sunjayBhatia
Copy link
Member

To connect the dots back in case anyone needs this info, see #11904 (comment)

We're seeing generally on Windows as well (the issue seems exacerbated there) that integration tests are not actually waiting for real time as fake/simulated time systems are in use in these tests

mattklein123 added a commit that referenced this issue Aug 7, 2020
@mattklein123
testing: fix multiple race conditions in simulated time tests
f01cc68 
Fixes #12480
Fixes #10568

Signed-off-by: Matt Klein <mklein@lyft.com>
@mattklein123 mattklein123 mentioned this issue Aug 7, 2020
Merged
mattklein123 added a commit that referenced this issue Aug 7, 2020
@mattklein123
testing: fix multiple race conditions in simulated time tests
6b59a9a 
Fixes #12480
Fixes #10568

Signed-off-by: Matt Klein <mklein@lyft.com>
mattklein123 added a commit that referenced this issue Aug 8, 2020
@mattklein123
testing: fix multiple race conditions in simulated time tests
3b90839 
Fixes #12480
Fixes #10568

Signed-off-by: Matt Klein <mklein@lyft.com>
mattklein123 added a commit that referenced this issue Aug 8, 2020
@mattklein123
testing: fix multiple race conditions in simulated time tests
9add2b7 
Fixes #12480
Fixes #10568

Signed-off-by: Matt Klein <mklein@lyft.com>
mattklein123 added a commit that referenced this issue Aug 9, 2020
@mattklein123
testing: fix multiple race conditions in simulated time tests
3d947c9 
Fixes #12480
Fixes #10568

Signed-off-by: Matt Klein <mklein@lyft.com>
mattklein123 added a commit that referenced this issue Aug 10, 2020
@mattklein123
testing: fix multiple race conditions in simulated time tests
30c6ef4 
Fixes #12480
Fixes #10568

Signed-off-by: Matt Klein <mklein@lyft.com>
mattklein123 added a commit that referenced this issue Aug 11, 2020
@mattklein123
testing: fix multiple race conditions in simulated time tests
af6e1fd 
Fixes #12480
Fixes #10568

Signed-off-by: Matt Klein <mklein@lyft.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@mattklein123 mattklein123

Labels
area/test flakes help wanted Needs help!
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

testing: fix multiple race conditions in simulated time tests envoyproxy/envoy
2 participants
@sunjayBhatia @mattklein123