Skip to content

Commit 0b30552

Browse files
committed
upstream: fix deadlock when destroying connections
When workers are enabled and a timeout occurs in a connection most of cases a deadlock is held in the active worker: ==1654992== Thread #4: Attempt to re-lock a non-recursive lock I already hold ==1654992== at 0x484BB44: ??? (in /usr/libexec/valgrind/vgpreload_helgrind-amd64-linux.so) ==1654992== by 0x197579: prepare_destroy_conn_safe (flb_upstream.c:435) ==1654992== by 0x197887: create_conn (flb_upstream.c:533) ==1654992== by 0x197DBB: flb_upstream_conn_get (flb_upstream.c:674) ==1654992== by 0x2396D3: http_post (http.c:86) ==1654992== by 0x23A5E5: cb_http_flush (http.c:338) ==1654992== by 0x17FE6B: output_pre_cb_flush (flb_output.h:511) ==1654992== by 0x503DAA: co_init (amd64.c:117) ==1654992== Lock was previously acquired ==1654992== at 0x484BC0F: ??? (in /usr/libexec/valgrind/vgpreload_helgrind-amd64-linux.so) ==1654992== by 0x19815F: flb_upstream_conn_timeouts (flb_upstream.c:780) ==1654992== by 0x17FEFC: cb_thread_sched_timer (flb_output_thread.c:58) ==1654992== by 0x193ED7: flb_sched_event_handler (flb_scheduler.c:422) ==1654992== by 0x180672: output_thread (flb_output_thread.c:265) ==1654992== by 0x199602: step_callback (flb_worker.c:44) ==1654992== by 0x484E8AA: ??? (in /usr/libexec/valgrind/vgpreload_helgrind-amd64-linux.so) ==1654992== by 0x4E3F926: start_thread (pthread_create.c:435) ==1654992== by 0x4ECF9E3: clone (clone.S:100) The following patch fix the behavior on prepare_destroy_conn_safe by 'trying to acquire' the mutex lock, if it fails to acquire it, it will asssume it's already locked and no new lock is required. Signed-off-by: Eduardo Silva <eduardo@calyptia.com>
1 parent cdb629d commit 0b30552

File tree

1 file changed

+6
-2
lines changed

1 file changed

+6
-2
lines changed

src/flb_upstream.c

+6-2
Original file line numberDiff line numberDiff line change
@@ -429,15 +429,19 @@ static int prepare_destroy_conn(struct flb_upstream_conn *u_conn)
429429
static inline int prepare_destroy_conn_safe(struct flb_upstream_conn *u_conn)
430430
{
431431
int ret;
432+
int locked = FLB_FALSE;
432433
struct flb_upstream *u = u_conn->u;
433434

434435
if (u->thread_safe == FLB_TRUE) {
435-
pthread_mutex_lock(&u->mutex_lists);
436+
ret = pthread_mutex_trylock(&u->mutex_lists);
437+
if (ret == 0) {
438+
locked = FLB_TRUE;
439+
}
436440
}
437441

438442
ret = prepare_destroy_conn(u_conn);
439443

440-
if (u->thread_safe == FLB_TRUE) {
444+
if (u->thread_safe == FLB_TRUE && locked) {
441445
pthread_mutex_unlock(&u->mutex_lists);
442446
}
443447

0 commit comments

Comments
 (0)