Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

SIGSEGV on OTLPAgent deletion when running test-nsolid-otlp-config-start.js #33

Closed
santigimeno opened this issue Nov 24, 2023 · 0 comments · Fixed by #34
Closed

SIGSEGV on OTLPAgent deletion when running test-nsolid-otlp-config-start.js #33

santigimeno opened this issue Nov 24, 2023 · 0 comments · Fixed by #34
Assignees
@santigimeno

Comments

@santigimeno
Copy link
Member

Observed on a resource-constrained box under load.

(lldb) bt all
* thread #1, name = 'nsolid', stop reason = signal SIGSEGV
  * frame #0: 0x00005633ec739ead nsolid`std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release(this=0x00007fa160004f00) at shared_ptr_base.h:168:16
    frame #1: 0x00005633ec738c8f nsolid`std::__shared_count<(__gnu_cxx::_Lock_policy)2>::~__shared_count(this=0x00007fa1600692b8) at shared_ptr_base.h:705:21
    frame #2: 0x00005633ee35f028 nsolid`std::__shared_ptr<opentelemetry::v1::ext::http::client::curl::HttpCurlGlobalInitializer, (__gnu_cxx::_Lock_policy)2>::~__shared_ptr(this=0x00007fa1600692b0) at shared_ptr_base.h:1154:7
    frame #3: 0x00005633ee35f048 nsolid`std::shared_ptr<opentelemetry::v1::ext::http::client::curl::HttpCurlGlobalInitializer>::~shared_ptr(this=nullptr) at shared_ptr.h:122:11
    frame #4: 0x00005633ee360ab8 nsolid`opentelemetry::v1::nostd::shared_ptr<opentelemetry::v1::ext::http::client::curl::HttpCurlGlobalInitializer>::shared_ptr_wrapper::~shared_ptr_wrapper(this=0x00007fa1600692a8) at shared_ptr.h:44:36
    frame #5: 0x00005633ee35ebe3 nsolid`opentelemetry::v1::nostd::shared_ptr<opentelemetry::v1::ext::http::client::curl::HttpCurlGlobalInitializer>::~shared_ptr(this=0x00007fa1600692a8) at shared_ptr.h:112:48
    frame #6: 0x00005633ee35a7d5 nsolid`opentelemetry::v1::ext::http::client::curl::HttpClient::~HttpClient(this=0x00007fa1600690e0) at http_client_curl.cc:161:1
    frame #7: 0x00005633ee36daa7 nsolid`void __gnu_cxx::new_allocator<opentelemetry::v1::ext::http::client::curl::HttpClient>::destroy<opentelemetry::v1::ext::http::client::curl::HttpClient>(this=0x00007fa1600690e0, __p=0x00007fa1600690e0) at new_allocator.h:168:13
    frame #8: 0x00005633ee36da27 nsolid`void std::allocator_traits<std::allocator<opentelemetry::v1::ext::http::client::curl::HttpClient> >::destroy<opentelemetry::v1::ext::http::client::curl::HttpClient>(__a=0x00007fa1600690e0, __p=0x00007fa1600690e0) at alloc_traits.h:535:15
    frame #9: 0x00005633ee36d8a5 nsolid`std::_Sp_counted_ptr_inplace<opentelemetry::v1::ext::http::client::curl::HttpClient, std::allocator<opentelemetry::v1::ext::http::client::curl::HttpClient>, (__gnu_cxx::_Lock_policy)2>::_M_dispose(this=0x00007fa1600690d0) at shared_ptr_base.h:528:35
    frame #10: 0x00005633ec739eb9 nsolid`std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release(this=0x00007fa1600690d0) at shared_ptr_base.h:168:16
    frame #11: 0x00005633ec738c8f nsolid`std::__shared_count<(__gnu_cxx::_Lock_policy)2>::~__shared_count(this=0x00007fa16001f0c8) at shared_ptr_base.h:705:21
    frame #12: 0x00005633ee343a8a nsolid`std::__shared_ptr<opentelemetry::v1::ext::http::client::HttpClient, (__gnu_cxx::_Lock_policy)2>::~__shared_ptr(this=0x00007fa16001f0c0) at shared_ptr_base.h:1154:7
    frame #13: 0x00005633ee343aaa nsolid`std::shared_ptr<opentelemetry::v1::ext::http::client::HttpClient>::~shared_ptr(this=nullptr) at shared_ptr.h:122:11
    frame #14: 0x00005633ee33fe20 nsolid`opentelemetry::v1::exporter::otlp::OtlpHttpClient::~OtlpHttpClient(this=0x00007fa16001f020) at otlp_http_client.cc:685:1
    frame #15: 0x00005633ecb72998 nsolid`std::default_delete<opentelemetry::v1::exporter::otlp::OtlpHttpClient>::operator(this=0x00007fa1600cb750, __ptr=0x00007fa16001f020)(opentelemetry::v1::exporter::otlp::OtlpHttpClient*) const at unique_ptr.h:85:2
    frame #16: 0x00005633ecb68b34 nsolid`std::unique_ptr<opentelemetry::v1::exporter::otlp::OtlpHttpClient, std::default_delete<opentelemetry::v1::exporter::otlp::OtlpHttpClient> >::~unique_ptr(this=0x7fa16001f020) at unique_ptr.h:361:17
    frame #17: 0x00005633ecb5ff10 nsolid`opentelemetry::v1::exporter::otlp::OtlpHttpExporter::~OtlpHttpExporter(this=0x00007fa1600cb6e0) at otlp_http_exporter.h:29:28
    frame #18: 0x00005633ecb5ff4c nsolid`opentelemetry::v1::exporter::otlp::OtlpHttpExporter::~OtlpHttpExporter(this=0x00007fa1600cb6e0) at otlp_http_exporter.h:29:28
    frame #19: 0x00005633ecb5ff85 nsolid`std::default_delete<opentelemetry::v1::exporter::otlp::OtlpHttpExporter>::operator(this=0x00005633f30d6790, __ptr=0x00007fa1600cb6e0)(opentelemetry::v1::exporter::otlp::OtlpHttpExporter*) const at unique_ptr.h:85:2
    frame #20: 0x00005633ecb5853a nsolid`std::unique_ptr<opentelemetry::v1::exporter::otlp::OtlpHttpExporter, std::default_delete<opentelemetry::v1::exporter::otlp::OtlpHttpExporter> >::~unique_ptr(this=0x7fa1600cb6e0) at unique_ptr.h:361:17
    frame #21: 0x00005633ecb4e452 nsolid`node::nsolid::otlp::OTLPAgent::~OTLPAgent(this=0x00005633f30d6120) at otlp_agent.cc:79:1
    frame #22: 0x00007fa17f0d8495 libc.so.6`__run_exit_handlers(status=0, listp=0x00007fa17f2ac838, run_list_atexit=true, run_dtors=true) at exit.c:113:8
    frame #23: 0x00007fa17f0d8610 libc.so.6`__GI_exit(status=<unavailable>) at exit.c:143:3
    frame #24: 0x00007fa17f0bcd97 libc.so.6`__libc_start_call_main(main=(nsolid`main at node_main.cc:96:34), argc=2, argv=0x00007fff22194d28) at libc_start_call_main.h:74:3
    frame #25: 0x00007fa17f0bce40 libc.so.6`__libc_start_main_impl(main=(nsolid`main at node_main.cc:96:34), argc=2, argv=0x00007fff22194d28, init=(_rtld_global), fini=<unavailable>, rtld_fini=<unavailable>, stack_end=0x00007fff22194d18) at libc-start.c:392:3
    frame #26: 0x00005633ec731355 nsolid`_start + 37
  thread #2, stop reason = signal 0
    frame #0: 0x0000000000000000
  thread #3, stop reason = signal 0
    frame #0: 0x0000000000000000
  thread #4, stop reason = signal 0
    frame #0: 0x0000000000000000
  thread #5, stop reason = signal 0
    frame #0: 0x0000000000000000
  thread #6, stop reason = signal 0
    frame #0: 0x0000000000000000
  thread #7, stop reason = signal 0
    frame #0: 0x0000000000000000
  thread #8, stop reason = signal 0
    frame #0: 0x00007fa17f1b8d58 libc.so.6`__GI_epoll_pwait(epfd=23, events=0x00007fa177ffbbd0, maxevents=1024, timeout=100, set=0x0000000000000000) at epoll_pwait.c:40:10
    frame #1: 0x00005633edf14d1f nsolid`uv__io_poll(loop=0x00005633f30e9048, timeout=100) at linux.c:1360:12
    frame #2: 0x00005633edef84c4 nsolid`uv_run(loop=0x00005633f30e9048, mode=UV_RUN_DEFAULT) at core.c:453:5
    frame #3: 0x00005633ecc1e6c8 nsolid`node::nsolid::EnvList::env_list_routine_((null)=0x00005633f30e94d8, envlist=0x00005633f30e9020) at nsolid_api.cc:1488:14
    frame #4: 0x00005633ecc330d2 nsolid`void nsuv::ns_thread::create_proxy_<void (*)(nsuv::ns_thread*, node::nsolid::EnvList*), node::nsolid::EnvList>(arg=0x00005633f30e94d8) at nsuv-inl.h:1829:6
    frame #5: 0x00007fa17f127ac3 libc.so.6`start_thread(arg=<unavailable>) at pthread_create.c:442:8
    frame #6: 0x00007fa17f1b8bf4 libc.so.6`__clone at clone.S:100
  thread #9, stop reason = signal 0
    frame #0: 0x00007fa17f124117 libc.so.6`__GI___futex_abstimed_wait_cancelable64 at futex-internal.c:57:12
    frame #1: 0x00007fa17f1240e9 libc.so.6`__GI___futex_abstimed_wait_cancelable64 at futex-internal.c:87:9
    frame #2: 0x00007fa17f1240e9 libc.so.6`__GI___futex_abstimed_wait_cancelable64(futex_word=0x00005633f30ea0e0, expected=0, clockid=<unavailable>, abstime=0x0000000000000000, private=<unavailable>) at futex-internal.c:139:10
    frame #3: 0x00007fa17f12fc78 libc.so.6`__new_sem_wait_slow64(sem=0x00005633f30ea0e0, abstime=<unavailable>, clockid=<unavailable>) at sem_waitcommon.c:183:10
    frame #4: 0x00005633edf0d508 nsolid`uv__sem_wait(sem=0x00005633f30ea0e0) at thread.c:639:9
    frame #5: 0x00005633edf0d67f nsolid`uv_sem_wait(sem=0x00005633f30ea0e0) at thread.c:695:5
    frame #6: 0x00005633ecc5ae8c nsolid`node::inspector::(anonymous namespace)::StartIoThreadMain(unused=0x0000000000000000) at inspector_agent.cc:85:16
    frame #7: 0x00007fa17f127ac3 libc.so.6`start_thread(arg=<unavailable>) at pthread_create.c:442:8
    frame #8: 0x00007fa17f1b8bf4 libc.so.6`__clone at clone.S:100
@santigimeno santigimeno self-assigned this Nov 24, 2023
santigimeno added a commit that referenced this issue Nov 24, 2023
@santigimeno
agents: fix crash in HttpCurlGlobalInitializer
7839d5a 
Make sure the HttpCurlGlobalInitializer static instance is created before
the OTLPAgent so it's destroyed after the OTLPAgent in __run_exit_handlers.
This fixes an insidious crash which happened if between the
HttpCurlGlobalInitializer and OTLPAgent destruction, a config update caused
the metrics_exporter_ to be reset to a specific exporter. This caused the
HttpCurlGlobalInitializer instance destructor to be called twice thus
crashing.

Fixes: #33
@santigimeno santigimeno mentioned this issue Nov 24, 2023
Merged
santigimeno added a commit that referenced this issue Dec 6, 2023
@santigimeno
agents: fix crash in HttpCurlGlobalInitializer
6ba5054 
Make sure the HttpCurlGlobalInitializer static instance is created before
the OTLPAgent so it's destroyed after the OTLPAgent in __run_exit_handlers.
This fixes an insidious crash which happened if between the
HttpCurlGlobalInitializer and OTLPAgent destruction, a config update caused
the metrics_exporter_ to be reset to a specific exporter. This caused the
HttpCurlGlobalInitializer instance destructor to be called twice thus
crashing.

Fixes: #33
PR-URL: #34
Reviewed-by: Trevor Norris <trev.norris@gmail.com>
trevnorris pushed a commit that referenced this issue Dec 7, 2023
@santigimeno @trevnorris
agents: fix crash in HttpCurlGlobalInitializer
55a2d6c 
Make sure the HttpCurlGlobalInitializer static instance is created before
the OTLPAgent so it's destroyed after the OTLPAgent in __run_exit_handlers.
This fixes an insidious crash which happened if between the
HttpCurlGlobalInitializer and OTLPAgent destruction, a config update caused
the metrics_exporter_ to be reset to a specific exporter. This caused the
HttpCurlGlobalInitializer instance destructor to be called twice thus
crashing.

Fixes: #33
PR-URL: #34
Reviewed-by: Trevor Norris <trev.norris@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@santigimeno santigimeno

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

agents: fix crash in HttpCurlGlobalInitializer nodesource/nsolid
1 participant
@santigimeno