TestSecurePairingSession fails on linux_x64_gcc_mbedtls, but only on some machines

[mspang]

Looks like it fails post main in static dtor:

yuif:27585:~/connectedhomeip% gdb out/debug/linux_x64_gcc_mbedtls/tests/TestSecurePairingSession
GNU gdb (Debian 9.2-1) 9.2
Copyright (C) 2020 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from out/debug/linux_x64_gcc_mbedtls/tests/TestSecurePairingSession...
(gdb) r
Starting program: /home/spang/connectedhomeip/out/debug/linux_x64_gcc_mbedtls/tests/TestSecurePairingSession 
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
'#0:','Test-CHIP-SecurePairing'
'#3:','WaitInit ','PASSED'
'#3:','Start    ','PASSED'
'#3:','Handshake','PASSED'
'#3:','Serialize','PASSED'
'#6:','0','4'
'#7:','0','29'

Program received signal SIGABRT, Aborted.
__GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
50	../sysdeps/unix/sysv/linux/raise.c: No such file or directory.
(gdb) bt
#0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
#1  0x00007ffff7bf9537 in __GI_abort () at abort.c:79
#2  0x0000555555597097 in chip::ReferenceCounted<chip::SecurePairingSessionDelegate, chip::DeleteDeletor<chip::SecurePairingSessionDelegate> >::Release (this=0x7fffffffd928) at ../../src/lib/core/ReferenceCounted.h:67
#3  0x0000555555595a15 in chip::SecurePairingSession::~SecurePairingSession (this=0x5555556180a0 <gTestPairingSession1>, __in_chrg=<optimized out>) at ../../src/transport/SecurePairingSession.cpp:54
#4  0x00007ffff7c125a7 in __run_exit_handlers (status=0, listp=0x7ffff7d92718 <__exit_funcs>, run_list_atexit=run_list_atexit@entry=true, run_dtors=run_dtors@entry=true) at exit.c:108
#5  0x00007ffff7c1274a in __GI_exit (status=<optimized out>) at exit.c:139
#6  0x00007ffff7bfacd1 in __libc_start_main (main=0x55555555b535 <main()>, argc=1, argv=0x7fffffffdc48, init=<optimized out>, fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7fffffffdc38) at ../csu/libc-start.c:342
#7  0x000055555555b47a in _start ()

@pan-apple

[issue-label-bot]

Issue-Label Bot is automatically applying the label bug to this issue, with a confidence of 0.96. Please mark this comment with 👍 or 👎 to give our bot feedback!

Links: app homepage, dashboard and code for this bot.

[pan-apple]

@mspang does it consistently fail on some machines? Do we have docker container where I can repro and test the fix?

[mspang]

It fails 100% of the time on one of my local linux machines, and 0% on another linux machine and my macbook.

Doesn't seem to fail on GitHub actions either.

[pan-apple]

I'll assign it to myself. I'll dig a bit more later today and tomorrow.
@mspang , if I give you a patch, would you be able to try it on your local machine where it consistently fails?

[mspang]

It's a use-after-free - the statically allocated SecurePairingSession tries to release a reference to a TestSecurePairingDelegate which was already de-allocated (it's on the stack, and we've already returned from main).

Accessing the refcount through the dangling pointer happens to read zero on the system that triggers the abort.

Fixed by #3357 (#3356 also makes the symptom go away, although we still have a dangling ptr - we just don't access it).