mbedtls 2.16.0 causes Open Broadcast Studio crash

[mcli]

Description

• Type: Bug
• Priority: Blocker
  On Fedora 29, mbedtls 2.16.0 causes Open Broadcast Studio to crash immediately when starting to stream.
  Works when using mbedtls 2.13.0. Core dump stack trace excerpt quoted below. Original issue is on Open Broadcast Studio forum: https://obsproject.com/forum/threads/start-streaming-causes-instant-crash-coredump.99615/

Core was generated by `obs --verbose'.
Program terminated with signal SIGABRT, Aborted.
#0 __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
50 ../sysdeps/unix/sysv/linux/raise.c: No such file or directory.
[Current thread is 1 (Thread 0x7f45a1839f00 (LWP 9738))]

Thread 1 (Thread 0x7f45a1839f00 (LWP 9738)):
#0 __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
set = {__val = {0, 140721941484536, 94858357099520, 94858395131264, 140721941484152, 139937174557660, 4, 0, 94858357442000, 206158430248, 140721941485760, 140721941485568, 94858357442000, 140721941484576, 140720308486148, 140721941484240}}
pid =
tid =
ret =
#1 0x00007f45a71a0895 in __GI_abort () at abort.c:79
save_stage = 1
act = {__sigaction_handler = {sa_handler = 0x8000000000000000, sa_sigaction = 0x8000000000000000}, sa_mask = {__val = {0 <repeats 14 times>, 140721941484560, 140721941484816}}, sa_flags = 4096, sa_restorer = 0x7ffc61559410}
sigs = {__val = {32, 0 <repeats 15 times>}}
#2 0x00007f45a71f9927 in __libc_message (action=action@entry=do_abort, fmt=fmt@entry=0x7f45a7307299 "%s\n") at ../sysdeps/posix/libc_fatal.c:181
ap = {{gp_offset = 24, fp_offset = 1607167915, overflow_arg_area = 0x7ffc61559520, reg_save_area = 0x7ffc615594b0}}
fd =
list =
nlist =
cp =
written =
#3 0x00007f45a720025c in malloc_printerr (str=str@entry=0x7f45a7309358 "malloc(): smallbin double linked list corrupted") at malloc.c:5382
No locals.
#4 0x00007f45a72035e4 in _int_malloc (av=av@entry=0x7f45a733ec60 <main_arena>, bytes=bytes@entry=384) at malloc.c:3640
tc_idx =
p =
nb = 400
idx = 25
bin = 0x7f45a733ee40 <main_arena+480>
victim =
size =
victim_index =
remainder =
remainder_size =
block =
bit =
map =
fwd =
bck =
tcache_unsorted_count =
tcache_nb =
tc_idx =
return_cached =
PRETTY_FUNCTION = "_int_malloc"
#5 0x00007f45a72058e6 in __libc_calloc (n=n@entry=1, elem_size=elem_size@entry=384) at malloc.c:3428
av =
oldtop = 0x5645f21cd130
p =
bytes = 384
sz = 384
csz =
oldtopsize = 921296
mem =
clearsize =
nclears =
d =
hook =
PRETTY_FUNCTION = "__libc_calloc"
#6 0x00007f456c6d6f04 in rsa_alloc_wrap () at /usr/src/debug/mbedtls-2.16.0-1.fc29.x86_64/library/pk_wrap.c:159
ctx =
#7 0x00007f456c6d67be in mbedtls_pk_setup (ctx=ctx@entry=0x5645f1088658, info=0x7f456c701ec0 <mbedtls_rsa_info>) at /usr/src/debug/mbedtls-2.16.0-1.fc29.x86_64/library/pk.c:142
No locals.
#8 0x00007f456c6d9d07 in mbedtls_pk_parse_subpubkey (p=p@entry=0x7ffc61559718, end=0x5645f212e466 "\243c0a0\035\006\003U\035\016\004\026\004\024R\330\210:\310\237xf\355\211\363{8p\224\311\002\002\066\320\060\017\006\003U\035\023\001\001\377\004\005\060\003\001\001\377\060\037\006\003U\035#\004\030\060\026\200\024R\330\210:\310\237xf\355\211\363{8p\224\311\002\002\066\320\060\016\006\003U\035\017\001\001\377\004\004\003\002\001\006\060\r\006\t*\206H\206\367\r\001\001\v\005", end@entry=0x5645f212e4cb "0\r\006\t*\206H\206\367\r\001\001\v\005", pk=pk@entry=0x5645f1088658) at /usr/src/debug/mbedtls-2.16.0-1.fc29.x86_64/library/pkparse.c:650
ret =
len = 526
alg_params = {tag = 5, len = 0, p = 0x5645f212e253 "\003\202\002\017"}
pk_alg = MBEDTLS_PK_RSA
pk_info =
#9 0x00007f456c693008 in x509_crt_parse_der_core (buflen=, buf=, crt=0x5645f1088510) at /usr/src/debug/mbedtls-2.16.0-1.fc29.x86_64/library/x509_crt.c:991
ret =
len = 107
end = 0x5645f212e4cb "0\r\006\t*\206H\206\367\r\001\001\v\005"
crt_end = 0x5645f212e6df ""
sig_params1 = {tag = 5, len = 0, p = 0x5645f212e146 "0k1\v0\t\006\003U\004\006\023\002IT1\016\060\f\006\003U\004\a\f\005Milan1#0!\006\003U\004\n\f\032Actalis S.p.A./033585209671'0%\006\003U\004\003\f\036Actalis Authentication Root CA0\036\027\r110922112202Z\027\r300922112202Z0k1\v0\t\006\003U\004\006\023\002IT1\016\060\f\006\003U\004\a\f\005Milan1#0!\006\003U\004\n\f\032Actalis S.p.A./03"...}
sig_oid2 = {tag = 0, len = 0, p = 0x0}
p = 0x5645f212e258 "0\202\002\n\002\202\002\001"
sig_params2 = {tag = 0, len = 0, p = 0x0}
ret =
len =
p =
end =
crt_end =
sig_params1 =
sig_params2 =
sig_oid2 =
#10 mbedtls_x509_crt_parse_der (buflen=, buf=, chain=) at /usr/src/debug/mbedtls-2.16.0-1.fc29.x86_64/library/x509_crt.c:1122
ret =
crt = 0x5645f1088510
prev = 0x5645f1ba5890
ret =
crt =
prev =
#11 mbedtls_x509_crt_parse_der (chain=0x5645f1b9ccf0, buf=, buflen=) at /usr/src/debug/mbedtls-2.16.0-1.fc29.x86_64/library/x509_crt.c:1089
ret =
crt = 0x5645f1b9ccf0
#12 0x00007f456c69398d in mbedtls_x509_crt_parse (chain=chain@entry=0x5645f1b9ccf0, buf=0x5645f1ce4ccb "\n# AddTrust External Root\n-----BEGIN CERTIFICATE-----\nMIIENjCCAx6gAwIBAgIBATANBgkqhkiG9w0BAQUFADBvMQswCQYDVQQGEwJTRTEU\nMBIGA1UEChMLQWRkVHJ1c3QgQUIxJjAkBgNVBAsTHUFkZFRydXN0IEV4dGVybmFs\nIFRUUCBOZXR3b3Jr"..., buflen=198630) at /usr/src/debug/mbedtls-2.16.0-1.fc29.x86_64/library/x509_crt.c:1219
use_len = 2083
ret =
pem = {buf = 0x5645f17a7e10 "0\202\005\273\060\202\003\243\240\003\002\001\002\002\bW\n\021\227B\304\343\314\060\r\006\t*\206H\206\367\r\001\001\v\005", buflen = 1471, info = 0x0}
success = 1
first_error = 0
total_failed = 0
#13 0x00007f456c693a84 in mbedtls_x509_crt_parse_file (chain=chain@entry=0x5645f1b9ccf0, path=path@entry=0x7ffc61559970 "/etc/ssl/certs//ca-bundle.crt") at /usr/src/debug/mbedtls-2.16.0-1.fc29.x86_64/library/x509_crt.c:1264
ret =
n = 205489
buf = 0x5645f1ce3200 "# ACCVRAIZ1\n-----BEGIN CERTIFICATE-----\nMIIH0zCCBbugAwIBAgIIXsO3pkN/pOAwDQYJKoZIhvcNAQEFBQAwQjESMBAGA1UE\nAwwJQUNDVlJBSVoxMRAwDgYDVQQLDAdQS0lBQ0NWMQ0wCwYDVQQKDARBQ0NWMQsw\nCQYDVQQGEwJFUzAeFw0xMTA1MDUwOT"...
#14 0x00007f456c693bb4 in mbedtls_x509_crt_parse_path (chain=chain@entry=0x5645f1b9ccf0, path=path@entry=0x7f456c755a70 "/etc/ssl/certs/") at /usr/src/debug/mbedtls-2.16.0-1.fc29.x86_64/library/x509_crt.c:1375
ret = 1
t_ret =
snp_ret =
sb = {st_dev = 64768, st_ino = 4719892, st_nlink = 1, st_mode = 33060, st_uid = 0, st_gid = 0, __pad0 = 0, st_rdev = 0, st_size = 205488, st_blksize = 4096, st_blocks = 408, st_atim = {tv_sec = 1548450668, tv_nsec = 573712143}, st_mtim = {tv_sec = 1540425561, tv_nsec = 920114261}, st_ctim = {tv_sec = 1547817973, tv_nsec = 700353567}, __glibc_reserved = {0, 0, 0}}
entry =
entry_name = "/etc/ssl/certs//ca-bundle.crt\000t.crt", '\000' <repeats 317 times>...
dir = 0x5645f1a4dce0
#15 0x00007f456c748b21 in RTMP_TLS_LoadCerts () at plugins/obs-outputs/librtmp/rtmp.c:341
chain = 0x5645f1b9ccf0
#16 0x00007f456c748bfb in RTMP_TLS_Init () at plugins/obs-outputs/librtmp/rtmp.c:374
pers = 0x7f456c755a80 "RTMP_TLS"
#17 0x00007f456c748e87 in RTMP_Init (r=r@entry=0x5645f1cd1070) at plugins/obs-outputs/librtmp/rtmp.c:536
No locals.
#18 0x00007f456c73f6d1 in rtmp_stream_create (settings=, output=0x5645eef63160) at plugins/obs-outputs/rtmp-stream.c:127
stream = 0x5645f1cd0f00
#19 0x00007f45a80e8a39 in obs_output_create (id=id@entry=0x5645f1bcfb40 "rtmp_output", name=name@entry=0x5645ed6a2e22 "adv_stream", settings=settings@entry=0x0, hotkey_data=hotkey_data@entry=0x0) at libobs/obs-output.c:152
info = 0x5645f0ee41f0
output = 0x5645eef63160
ret =
#20 0x00005645ed61fd2f in AdvancedOutput::StartStreaming (this=0x5645ee624b80, service=0x5645f1fb8040) at UI/window-basic-main-outputs.cpp:1443
codec =
trackIndex = 1
type = 0x5645f1bcfb40 "rtmp_output"
reconnect =
retryDelay =
maxRetries =
useDelay =
delaySec =
preserveDelay =
bindIP =
enableNewSocketLoop =
enableLowLatencyMode =
settings =
error =
has_last_error =
#21 0x00005645ed5aef0a in OBSBasic::StartStreaming() () at /usr/include/c++/8/bits/unique_ptr.h:342
copiedTransformInfo = {pos = {{{x = 0, y = 0}, ptr = {0, 0}}}, rot = 0, scale = {{{x = 0, y = 0}, ptr = {0, 0}}}, alignment = 0, bounds_type = OBS_BOUNDS_NONE, bounds_alignment = 0, bounds = {{{x = 0, y = 0}, ptr = {0, 0}}}}
copiedCropInfo = {left = 0, top = 0, right = 0, bottom = 0}
scaled_vals = {1, 1.25, 1.3333333333333333, 1.5, 1.6666666666666667, 1.75, 2, 2.25, 2.5, 2.75, 3, 0}

OS
Linux - Fedora 29

mbed TLS build:
Version: 2.16.0
OS version: 4.20.5
Configuration: Fedora 29 x86_64
Compiler and options:
Precompiled binary from Fedora 29 update

Additional environment information:

Peer device TLS stack and version
Other - Open Broadcast Studio
Version: 22.0.3

Expected behavior
OBS should not crash when starting to stream.
Actual behavior
crashes (see stack trace above)
Steps to reproduce
Run Open Broadcast Studio and start streaming

[RonEld]

@mcli Thank you for reporting this!
Have you happened to use a debugger to understand root cause?
From stack trace, it seems the crash happens from a heap allocation call. I assume you verified mbedtls_calloc refers to the correct calloc function of your platform.
We will look into this

[ciarmcom]

ARM Internal Ref: IOTSSL-2761

[RobertBeekmans]

Unfortunately I can confirm mbedTLS v2.16.0 also causes a crash within my application running on a STM32F7. Going back to v2.13.0 (as I had previously) did NOT trigger the same issue.
In my opinion it is caused by a buffer overrun (and/or stack corruption)...
I did not yet try mbedTLS v2.14.1 yet.

[RonEld]

@RobertBeekmans Thank you for your input!
Could you share additional information? For example, what is the flow? Do you have stack trace?
We are not being able to reproduce this yet, so any additional input would assist.

[RobertBeekmans]

I'm using mbedTLS with the following as a client within our embedded application:

• Keil MDK-ARM Essential (µVision V5.26.2.0 and C compiler V5.06 update 6 (build 750))
• Cipher suite TLS-ECDHE-ECDSA-WITH-AES-256-GCM-SHA384
• lwIP v2.0.3 TCP/IP stack
• Hardware acceleration (see mbed OS for STMF7 implementation)

After a successful TLS handshake, data is being transferred between client and server and sometimes data of the FLASH file system get corrupted (results in not being able to search for files).
Either data in one of the RAM sections is "touched" or some data on the stack is "touched".
This only is the case with v2.16.0 and not when using the same application with v2.13.0.
I don't have a stack trace, nevertheless I hope this bit of information helps you.
Only 1 of 5 times the application is successfully completing the synchronization between server and client, so > 75% of my attempts failed with v2.16.0.

[RonEld]

According to the log" malloc(): smallbin double linked list corrupted" and additional input from @RobertBeekmans , I am assuming there is some sort of memory overflow in the system. Unfortunately, we still haven't managed to reproduce this.

I am trying to narrow down the versions. Does this crash happen on version 2.14.1 and \ or 2.15.1 ( the latter was not release as a standalone release, but for Mbed OS, but it could be used as well for the purpose of reproduction)?

[Patater]

Just an idea, but we removed some checks on parameters that the caller should be responsible for ensuring are correct before calling. Perhaps some of the parameters passed to Mbed TLS are invalid and not getting caught or ignored as in previous versions of Mbed TLS. Could you try to reproduce the issue with MBEDTLS_CHECK_PARAMS set in your config.h? You'll also need to implement your own mbedtls_param_failed() function or MBEDTLS_PARAM_FAILED() macro.

An excerpt from the documentation for this feature:

 * When this flag is defined, the library additionally attempts to validate
 * parameters that are fully controlled by the application, and should always
 * be valid if the application code is fully correct and trusted.

[mcli]

Hi All,

I finally got around to running obs in a debugger using the debuginfo rpms. I haven't yet got around to compiling mbedtls yet, but maybe we can make some progress looking at the initialized structure.
The program crashes when loading the certs. Shortly before that the RTMP_TLS_ctx gets initialized:

const char * pers = "RTMP_TLS";
RTMP_TLS_ctx = calloc(1,sizeof(struct tls_ctx));

mbedtls_ssl_config_init(&RTMP_TLS_ctx->conf);
mbedtls_ctr_drbg_init(&RTMP_TLS_ctx->ctr_drbg);
mbedtls_entropy_init(&RTMP_TLS_ctx->entropy);

mbedtls_ctr_drbg_seed(&RTMP_TLS_ctx->ctr_drbg,
                      mbedtls_entropy_func,
                      &RTMP_TLS_ctx->entropy,
                      (const unsigned char *)pers,
                      strlen(pers));

RTMP_TLS_LoadCerts();

Here is the RTMP_TLS_ctx structure contents (conf expanded) after the initialization sequence. I see that there is some uninitialized memory (psk_len, psk_identity_len). Let me know if there is anything else you'd like to look at.

RTMP_TLS_ctx TLS_CTX
1: [-] entropy mbedtls_entropy_context
2: [?] accumulator_started 1
2: [+] accumulator mbedtls_sha512_context
2: [?] source_count 3
2: [+] source mbedtls_entropy_source_state [20]
2: [+] mutex mbedtls_threading_mutex_t
1: [+] ctr_drbg mbedtls_ctr_drbg_context
1: [-] conf mbedtls_ssl_config
2: [+] ciphersuite_list const int *[4]
2: [?] f_dbg 0x5a498179cf89bb71
2: [?] p_dbg 0x503e84282ec34950
2: [?] f_rng 0x2ae51924a1f7e232
2: [?] p_rng 0x48334c58dcb3ce33
2: [?] f_get_cache 0xe6c7ae345405fc23
2: [?] f_set_cache 0x9d8c1a2ed62ccb4
2: [?] p_cache 0x32c094655f60af40
2: [?] f_sni 0x4b5f264d93cb0e5d
2: [?] p_sni 0xafa0167c3283bc1a
2: [?] f_vrfy 0x7b45ea62d48ec0c8
2: [?] p_vrfy 0x5f0efdf70d519ca2
2: [?] f_psk 0x90a66d7dc956e919
2: [?] p_psk 0xb84aa71afc63ca26
2: [?] f_cookie_write 0x600924488c4db885
2: [?] f_cookie_check 0xdf379c0985cfb3e6
2: [?] p_cookie 0x80f0415cfdfbbfe8
2: [?] f_ticket_write 0xad33bd8cff1e539f
2: [?] f_ticket_parse 0x226d47cd9bfff5fa
2: [?] p_ticket 0x630105dce328de3e
2: [?] f_export_keys 0x477fcd1d65ad52d5
2: [?] p_export_keys 0x1055f4fe1afba7b2
2: [+] cert_profile const mbedtls_x509_crt_profile * 0x0
2: [+] key_cert mbedtls_ssl_key_cert * 0x0
2: [+] ca_chain mbedtls_x509_crt * 0x0
2: [+] ca_crl mbedtls_x509_crl * 0x0
2: [+] sig_hashes const int * 0x0
2: [+] curve_list const mbedtls_ecp_group_id * 0x0
2: [+] dhm_P mbedtls_mpi
2: [+] dhm_G mbedtls_mpi
2: [?] psk 0x0
2: [?] psk_len 16924626128907405157
2: [?] psk_identity 0x0
2: [?] psk_identity_len 4703010428147742877
2: [+] alpn_list const char ** 0x0
2: [?] read_timeout 814252014
2: [?] hs_timeout_min 1387587976
2: [?] hs_timeout_max 1758607326
2: [?] renego_max_records 1415847117
2: [+] renego_period unsigned char [8]
2: [?] badmac_limit 3834419145
2: [?] dhm_min_bitlen 3052819520
2: [?] max_major_ver 207 '\317'
2: [?] max_minor_ver 9 '\t'
2: [?] min_major_ver 102 'f'
2: [?] min_minor_ver 133 '\205'
2: [?] endpoint 1
2: [?] transport 1
2: [?] authmode 1
2: [?] allow_legacy_renegotiation 1
2: [?] arc4_disabled 1
2: [?] mfl_code 0
2: [?] encrypt_then_mac 1
2: [?] extended_ms 0
2: [?] anti_replay 0
2: [?] cbc_record_splitting 1
2: [?] disable_renegotiation 0
2: [?] trunc_hmac 1
2: [?] session_tickets 0
2: [?] fallback 1
2: [?] cert_req_ca_list 1
1: [+] ssn mbedtls_ssl_session
1: [+] cacert mbedtls_x509_crt * 0x0
1: [-] net mbedtls_net_context
2: [?] fd -1857207025

[RonEld]

@mcli Thank you for your information.
May I know what does RTMP_TLS_LoadCerts() do? Does it load certificates to context and verifies them?
Have you set the ca root certificate with mbedtls_ssl_conf_ca_chain()?
I see that cacert has not been set, so I suspect that since you have a crash while loading certificates, it could be a hint.
The static functionx509_crt_find_parent() has changed since version 2.13, so perhaps sending a NULL trust_ca causes the issue.
cc @mpg

[mcli]

@RonEld Here is the beginning portion of RTMP_TLS_LoadCerts(). The program crashes on the calloc. The call to mbedtls_ssl_conf_ca_chain() is at the end of the function. It never gets there.

void
RTMP_TLS_LoadCerts() {
#ifdef USE_MBEDTLS
mbedtls_x509_crt *chain = RTMP_TLS_ctx->cacert = calloc(1, sizeof(struct mbedtls_x509_crt));
mbedtls_x509_crt_init(chain);

[mcli]

@RonEld, @mpg
Here's some more info. When I step through the program, it crashes on the calloc line in RTMP_TLS_LoadCerts(). However, if I don't set the break point, the backtrace indicates that the last mbedtls function crashes at: mbedtls_x509_crt_parse_path (chain=chain@entry=0x555556c27aa0, path=path@entry=0x7fffa5e2ea70 "/etc/ssl/certs/") at /usr/src/debug/mbedtls-2.16.0-1.fc29.x86_64/library/x509_crt.c:1341

Here's the complete back trace:

#0 0x00007ffff533e53f in raise () from /lib64/libc.so.6
#1 0x00007ffff5328895 in abort () from /lib64/libc.so.6
#2 0x00007ffff5381927 in __libc_message () from /lib64/libc.so.6
#3 0x00007ffff538825c in malloc_printerr () from /lib64/libc.so.6
#4 0x00007ffff538b1d4 in _int_malloc () from /lib64/libc.so.6
#5 0x00007ffff538cc8a in malloc () from /lib64/libc.so.6
#6 0x00007ffff53caaca in __alloc_dir () from /lib64/libc.so.6
#7 0x00007ffff53cabcd in opendir_tail () from /lib64/libc.so.6
#8 0x00007fffa5d42af8 in mbedtls_x509_crt_parse_path (chain=chain@entry=0x555556c27aa0, path=path@entry=0x7fffa5e2ea70 "/etc/ssl/certs/") at /usr/src/debug/mbedtls-2.16.0-1.fc29.x86_64/library/x509_crt.c:1341
#9 0x00007fffa5e21b21 in RTMP_TLS_LoadCerts () at plugins/obs-outputs/librtmp/rtmp.c:341
#10 0x00007fffa5e21bfb in RTMP_TLS_Init () at plugins/obs-outputs/librtmp/rtmp.c:374
#11 0x00007fffa5e21e87 in RTMP_Init (r=r@entry=0x555556c44210) at plugins/obs-outputs/librtmp/rtmp.c:536
#12 0x00007fffa5e186d1 in rtmp_stream_create (settings=, output=0x555556945520) at plugins/obs-outputs/rtmp-stream.c:127
#13 0x00007ffff6270a39 in obs_output_create (id=id@entry=0x55555692a6a0 "rtmp_output", name=name@entry=0x555555729dcc "simple_stream", settings=settings@entry=0x0, hotkey_data=hotkey_data@entry=0x0) at libobs/obs-output.c:152
#14 0x00005555556a61bf in SimpleOutput::StartStreaming (this=0x555556871910, service=0x5555565db840) at UI/window-basic-main-outputs.cpp:665
#15 0x0000555555635f0a in OBSBasic::StartStreaming() () at /usr/include/c++/8/bits/unique_ptr.h:342
#16 0x00005555556363e8 in OBSBasic::on_streamButton_clicked() () at UI/window-basic-main.cpp:5264
#17 0x000055555570d389 in OBSBasic::qt_static_metacall (_o=0x5555559adee0, _c=, _id=, _a=0x7fffffffc180) at UI/obs_autogen/EWIEGA46WW/moc_window-basic-main.cpp:923
#18 0x000055555570feb3 in OBSBasic::qt_metacall (this=0x5555559adee0, _c=QMetaObject::InvokeMetaMethod, _id=122, _a=0x7fffffffc180) at UI/obs_autogen/EWIEGA46WW/moc_window-basic-main.cpp:1029
#19 0x00007ffff5aa1147 in QMetaObject::activate(QObject*, int, int, void**) () from /lib64/libQt5Core.so.5
#20 0x00007ffff7b815b6 in QAbstractButton::clicked(bool) () from /lib64/libQt5Widgets.so.5
#21 0x00007ffff7b817de in ?? () from /lib64/libQt5Widgets.so.5
#22 0x00007ffff7b82c33 in ?? () from /lib64/libQt5Widgets.so.5
#23 0x00007ffff7b82e05 in QAbstractButton::mouseReleaseEvent(QMouseEvent*) () from /lib64/libQt5Widgets.so.5
#24 0x00007ffff7ad8378 in QWidget::event(QEvent*) () from /lib64/libQt5Widgets.so.5
#25 0x00007ffff7a99285 in QApplicationPrivate::notify_helper(QObject*, QEvent*) () from /lib64/libQt5Widgets.so.5
#26 0x00007ffff7aa0be8 in QApplication::notify(QObject*, QEvent*) () from /lib64/libQt5Widgets.so.5
#27 0x00007ffff5a78ec6 in QCoreApplication::notifyInternal2(QObject*, QEvent*) () from /lib64/libQt5Core.so.5
#28 0x00007ffff7a9fedd in QApplicationPrivate::sendMouseEvent(QWidget*, QMouseEvent*, QWidget*, QWidget*, QWidget**, QPointer&, bool) () from /lib64/libQt5Widgets.so.5
#29 0x00007ffff7af3118 in ?? () from /lib64/libQt5Widgets.so.5
#30 0x00007ffff7af5cbe in ?? () from /lib64/libQt5Widgets.so.5
#31 0x00007ffff7a99285 in QApplicationPrivate::notify_helper(QObject*, QEvent*) () from /lib64/libQt5Widgets.so.5
#32 0x00007ffff7aa09a0 in QApplication::notify(QObject*, QEvent*) () from /lib64/libQt5Widgets.so.5
#33 0x00007ffff5a78ec6 in QCoreApplication::notifyInternal2(QObject*, QEvent*) () from /lib64/libQt5Core.so.5
#34 0x00007ffff5e194d3 in QGuiApplicationPrivate::processMouseEvent(QWindowSystemInterfacePrivate::MouseEvent*) () from /lib64/libQt5Gui.so.5
#35 0x00007ffff5e1b5d5 in QGuiApplicationPrivate::processWindowSystemEvent(QWindowSystemInterfacePrivate::WindowSystemEvent*) () from /lib64/libQt5Gui.so.5
#36 0x00007ffff5df670b in QWindowSystemInterface::sendWindowSystemEvents(QFlagsQEventLoop::ProcessEventsFlag) () from /lib64/libQt5Gui.so.5
#37 0x00007fffe294e85f in ?? () from /lib64/libQt5XcbQpa.so.5
#38 0x00007ffff5a77e0b in QEventLoop::exec(QFlagsQEventLoop::ProcessEventsFlag) () from /lib64/libQt5Core.so.5
#39 0x00007ffff5a7fed6 in QCoreApplication::exec() () from /lib64/libQt5Core.so.5
#40 0x00005555555f9598 in run_program (argv=0x7fffffffd6f8, argc=, logFile=...) at UI/obs-app.cpp:1732
#41 main (argc=, argv=0x7fffffffd6f8) at UI/obs-app.cpp:2286

[hanno-becker]

Hi @mcli,

thank you for the information! As mentioned earlier, it looks like there is some heap corruption, e.g. due to a buffer overflow or double free. Alternatively to @Patater's suggestion of turning on parameter validation: Can you compile the library with an address sanitizer enabled to see where the (likely) corruption is happening?

Regards,
Hanno

[mcli]

Hi All,

I traced the problem to mbedtls_x509_crt_parse_file returning MBEDTLS_ERR_X509_CERT_UNKNOWN for /etc/ssl/ca-bundle.trust.crt. mbedtls_x509_crt_parse_file then regards it as a partial success. OBS is treating all non-zero return values as errors and then setting RTMP_TLS_ctx->cacert=null.
Certainly there are issues with the calling code as it should not consider the non-zero return value to be an error. However, IMHO embedtls should be handling the .trust.crt file in a better/proper manner and return a 0 for success (not partial success). As proof of concept fix, I did a commit on my fork to just ignore ".trust.crt" files in the cert dir. (see the commit above). I then ran obs and streaming no longer crashed.

I'm interested in soliciting your opinions as to what the proper fix should be. I'm willing to code it along with a test for a pull request.

[RonEld]

Hi @mcli
Thank you for your information.
If I understand your description correctly, then the API behaves as described and expected, and it's only a platform issue, that it treats the non zero return code as error. Am I right?
If this is the case, then you are requesting a new feature, to change the API, to return different error code. ( perhaps an out parameter stating number of failed certificates? )

It is strange though why this didn't happen on mbedtls-2.13.0. Probably need to dig deeper in the ChangeLog.
Is there a way you can handle this return code in your application?

[RonEld]

Since to my understanding, this is a feature request, and not a bug, changing the label to "enhancement"

[RobertBeekmans]

I can confirm mbedTLS v2.16.1 runs fine now with my application running on a STM32F7 (and it uses the STM32 HW crypto accell as implemented within mbed OS for AES, MD5, SHA1 and SHA256).

[RonEld]

@RobertBeekmans Thank you for your information

[davidhorstmann-arm]

Looking closely at the stacktrace and description, this seems to be a duplicate of #3005, which was fixed on the OBS side by obsproject/obs-studio@4d89123.

I'll close this as fixed elsewhere.