sssd crashes with hardened_malloc in ld preload

[mike2307]

sssd on fedora 31 reports a segementation fault when running with lib hardened malloc.

Package version: sssd-common-2.2.2-1.fc31
Crashing function: ldb_kv_dn_list_load

Stack trace:

Process 5945 (sssd) of user 0 dumped core.
Stack trace of thread 5945: #0 0x00007f50329388b0 _int_free (libc.so.6)
#1 0x000072d6d7852f0d ldb_kv_dn_list_load (libldb-key-value.so)
#2 0x000072d6d7854f24 ldb_kv_index_add1 (libldb-key-value.so)
#3 0x000072d6d7856cce ldb_kv_index_add_all (libldb-key-value.so)
#4 0x000072d6d785714e ldb_kv_index_add_new (libldb-key-value.so)
#5 0x000072d6d784fcb9 ldb_kv_add_internal (libldb-key-value.so)
#6 0x000072d6d7850a58 ldb_kv_callback (libldb-key-value.so)
#7 0x00007f5032a96f3d tevent_common_invoke_timer_handler (libtevent.so.0)
#8 0x00007f5032a970e0 tevent_common_loop_timer_delay (libtevent.so.0)
#9 0x00007f5032a981bc epoll_event_loop_once (libtevent.so.0)
#10 0x00007f5032a9653b std_event_loop_once (libtevent.so.0)
#11 0x00007f5032a915d8 _tevent_loop_once (libtevent.so.0)
#12 0x00007f5032bed47b ldb_wait (libldb.so.2)
#13 0x00007f5032bee033 ldb_autotransaction_request (libldb.so.2)
#14 0x00007f5032beee45 ldb_add (libldb.so.2)
#15 0x0000558a4ad704c1 confdb_setup (sssd)
#16 0x0000558a4ad6dc11 load_configuration (sssd)
#17 0x0000558a4ad69f33 main (sssd)
#18 0x00007f50328d61a3 __libc_start_main (libc.so.6)
#19 0x0000558a4ad6a80e _start (sssd) 

[mike2307]

The corresponding bugzilla ticket is https://bugzilla.redhat.com/show_bug.cgi?id=1761232

[thestinger]

Most issues like this are upstream memory corruption bugs uncovered by hardened_malloc. It uncovers latent memory corruption bugs as part of mitigating exploitation. However, in this case, I see _int_free in the backtrace which indicates that glibc malloc is being at least partially used. I think something unusual is happening here. The malloc implementation isn't being properly substituted via the mechanism you're using, perhaps due to something odd about how the software is compiled / linked. Even though glibc officially supports using linker preloading for malloc replacement, there are edge cases that aren't supported properly by them. I'm curious if you encounter a crash using jemalloc or another malloc implementation via the same replacement mechanism that you're using.

[mike2307]

When using jemalloc, sssd works. I'm getting some SELinux errors regarding unix_chkpwd instead...

[mike2307]

In the RPM spec file of the failing library, I see a dependency to library called 'talloc'. See libldb.spec.txt.
Maybe that's related somehow...

[mike2307]

By the way; This is the method I'm using to replace the malloc/free calls:

# cat /etc/ld.so.preload
/usr/local/lib64/libhardened_malloc.so

The library is compiled with all settings to their defaults, using the latest version on master.

[mike2307]

I ran the application with

LD_PRELOAD=/usr/local/lib64/libhardened_malloc.so LD_DEBUG=all sssd 2> hardened.txt
LD_PRELOAD=/usr/local/lib64/libjemalloc.so LD_DEBUG=all sssd 2> jemalloc.txt

The output is attached sssdtest.tar.gz.

The interesting part starts at hardened.txt, line 134592. From that point onwards, libhardened_malloc is not considered any more when looking up symbols.
The matching section in jemalloc.txt start at line 138061. libjemalloc however is used for resolving symbols just fine afterwards...

Edit:
Seems like i misaligned the two files. Actually the look quite similar. But one works; the other not. Maybe the debug output contains some hints elsewhere...

[thestinger]

@mike2307 Is this still an issue?

[mike2307]

@thestinger
Yes, this issue is still valid.

[thestinger]

@mike2307 Can you try with v6 of hardened_malloc? There have been various little fixes over time since I last asked.

[mike2307]

@thestinger:
No, sorry. The issue is still present with the latest version.