Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Using the parcelport TCP causes the memory usage to continuously increase, and I'm not sure if it is a memory leak #6574

Open
phil-skillwon opened this issue Nov 17, 2024 · 4 comments
Open

Using the parcelport TCP causes the memory usage to continuously increase, and I'm not sure if it is a memory leak #6574

phil-skillwon opened this issue Nov 17, 2024 · 4 comments
Labels
category: parcel transport type: defect

Comments

@phil-skillwon
Copy link

Recently, I’ve been validating the feasibility of using HPX as the foundational framework for our team's signal processing algorithm development. However, during testing, I noticed what seems to be a memory leak issue with HPX.

So, I wrote a separate test program using the TCP parcelport to test data interaction across multiple nodes and discovered what looked like a memory leak. But I’m not entirely sure, so I’m seeking your help here.

I used two nodes, running on two different hosts (Ubuntu 22.04 LTS), and the test code is as follows:

static vector<std::byte> getData(const size_t sz) 
{
    vector<std::byte> data(sz, (std::byte)(0xFF));
    
    return data;
}

HPX_PLAIN_ACTION(getData, GetDataAction);

int hpx_main(int argc, char* argv[])
{
    hpx::error_code ec = hpx::make_success_code();
    std::vector<hpx::id_type> localities = hpx::find_all_localities(ec);
    if (hpx::error::success != ec.value()) 
    {
        printf("find_all_localities executed failed, %s\n", ec.get_message().c_str());
        return -1;
    }

    if (localities.size() < 2) 
    {
        printf("this program requires at least 2 localities\n");
        return -2;
    }

    printf("num of localities: %ld\n", localities.size());
    for (const auto& loc : localities) 
    {
        hpx::naming::gid_type gid = loc.get_gid();
        std::string address = hpx::get_locality_name(loc).get();
        std::uint32_t localityId = hpx::naming::get_locality_id_from_gid(gid);

        printf("locality id: %d\n", localityId);
        printf("locality name: %s, id: %08X\n", address.c_str(), localityId);
    }

    size_t dataSize = 960256;
    while (true) 
    {
        hpx::this_thread::sleep_for(1400us);

        auto dataNode0 = hpx::sync<GetDataAction>(localities[0], dataSize);
        auto dataNode1 = hpx::sync<GetDataAction>(localities[1], dataSize);

        printf("node0, data size: %ld, node1, data size: %ld\n", dataNode0.size(), dataNode1.size());
    }
    
    return hpx::finalize();
}

Node 0 is the root node. I observed the memory usage on both Node 0 and Node 1. Both hosts have 8GB of physical memory.

When the test program started, the memory usage on both nodes was about 0.4%. But after 1 hour, the memory usage on Node 0 increased to 0.7%, while the memory usage on Node 1 remained at 0.4%.

After about 24 hours, the memory usage on Node 0 reached 1.9%, while the memory usage on Node 1 remained at 0.4%. This looks like a memory leak.

Later, I modified the test code to run a single process on one host, and there was no increase in memory usage.

However, my test code is extremely simple, so it’s unlikely that the issue is due to my code. Could you help analyze this problem?
@hkaiser
Copy link
Member

hkaiser commented Nov 17, 2024

@phil-skillwon Could you compile your test code (and possibly HPX) with -DHPX_WITH_SANITIZERS=On, please? This should report memory leaks, if any. I'd be more than happy to assist in diagnosing and fixing those.

@phil-skillwon
Copy link
Author

@hkaiser
I referred to the official documentation: https://hpx-docs.stellar-group.org/latest/html/manual/debugging_hpx_applications.html?highlight=hpx_with_sanitizers

Using sanitizers with HPX applications

Warning

Not all parts of HPX are sanitizer clean. This means that users may end up with false positives from HPX itself when using sanitizers for their applications.

To use sanitizers with HPX, turn on HPX_WITH_SANITIZERS and turn off HPX_WITH_STACKOVERFLOW_DETECTION during CMake configuration. It’s recommended to also build Boost with the same sanitizers that will be used for HPX. The appropriate sanitizers can then be enabled using CMake by appending -fsanitize=address -fno-omit-frame-pointer to CMAKE_CXX_FLAGS and -fsanitize=address to CMAKE_EXE_LINKER_FLAGS. Replace address with the sanitizer that you want to use.

Then, I recompiled HPX and my test code:
Rebuild HPX:

cmake .. -DCMAKE_BUILD_TYPE=RelWithDebInfo -DHPX_WITH_STATIC_LINKING=OFF -DBUILD_SHARED_LIBS=ON -DHPX_WITH_SANITIZERS=ON -DHPX_WITH_STACKOVERFLOW_DETECTION=OFF -DHPX_WITH_PARCELPORT_TCP=ON -DHPX_WITH_MALLOC=system -DBoost_ROOT=/home/skillwon/Work/3rdParty/boost_1_84_0 -DAsio_ROOT=/home/skillwon/Work/3rdParty/asio-1.28.1 -DHwloc_ROOT=/home/skillwon/Work/3rdParty/hwloc-2.10.0 -DCMAKE_INSTALL_PREFIX=/usr/local/lib/Skillwon/Hpx

My test code:

set(CMAKE_CXX_FLAGS "-g -O0 -std=c++${CMAKE_CXX_STANDARD} -v -Wl,--verbose -Wall -fPIC -fsanitize=address -fno-omit-frame-pointer ${CMAKE_CXX_FLAGS} -lasan -DHPX_WITH_SANITIZERS=ON -Wno-comment")
set(CMAKE_EXE_LINKER_FLAGS "${CMAKE_EXE_LINKER_FLAGS} -fsanitize=address")

ASAN env:

export ASAN_OPTIONS=halt_on_error=0:use_sigaltstack=0:detect_leaks=1:malloc_context_size=15:log_path=/home/skillwon/Test/bin/debug/Asan.log

I ran the tests again. During the test, I found that only the root node outputs ASAN debug information, while the other worker node does not output any debug information. Below is a portion of the address sanitizer debug output:

...
...
...
=================================================================
==864844==ERROR: LeakSanitizer: detected memory leaks

Direct leak of 27648 byte(s) in 432 object(s) allocated from:
    #0 0x7fd08a1ab3f5 in __interceptor_aligned_alloc ../../../../src/libsanitizer/asan/asan_malloc_linux.cc:183
    #1 0x7fd0897255ce in hpx::lockfree::queue<hpx::threads::thread_data*, hpx::util::aligned_allocator<hpx::threads::thread_data*, std::allocator<hpx::threads::thread_data*> >, 0ul, false>::queue(unsigned long) /home/skillwon/Work/3rdParty/hpx-1.10.0/libs/core/allocator_support/include/hpx/allocator_support/aligned_allocator.hpp:50
    #2 0x7fd0897255ce in hpx::threads::policies::lockfree_fifo_backend<hpx::threads::thread_data*>::lockfree_fifo_backend(unsigned long, unsigned long) /home/skillwon/Work/3rdParty/hpx-1.10.0/libs/core/schedulers/include/hpx/schedulers/lockfree_queue_backends.hpp:57
    #3 0x7fd0897255ce in hpx::threads::policies::thread_queue<std::mutex, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_fifo>::thread_queue(hpx::threads::policies::thread_queue_init_parameters const&) /home/skillwon/Work/3rdParty/hpx-1.10.0/libs/core/schedulers/include/hpx/schedulers/thread_queue.hpp:499

Direct leak of 2112 byte(s) in 33 object(s) allocated from:
    #0 0x7fd08a1ab3f5 in __interceptor_aligned_alloc ../../../../src/libsanitizer/asan/asan_malloc_linux.cc:183
    #1 0x7fd0897254ae in hpx::lockfree::queue<hpx::threads::detail::thread_data_reference_counting*, hpx::util::aligned_allocator<hpx::threads::detail::thread_data_reference_counting*, std::allocator<hpx::threads::detail::thread_data_reference_counting*> >, 0ul, false>::queue(unsigned long) /home/skillwon/Work/3rdParty/hpx-1.10.0/libs/core/allocator_support/include/hpx/allocator_support/aligned_allocator.hpp:50
    #2 0x7fd0897254ae in hpx::threads::policies::lockfree_fifo_backend<hpx::threads::detail::thread_data_reference_counting*>::lockfree_fifo_backend(unsigned long, unsigned long) /home/skillwon/Work/3rdParty/hpx-1.10.0/libs/core/schedulers/include/hpx/schedulers/lockfree_queue_backends.hpp:57
    #3 0x7fd0897254ae in hpx::threads::policies::thread_queue<std::mutex, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_fifo>::thread_queue(hpx::threads::policies::thread_queue_init_parameters const&) /home/skillwon/Work/3rdParty/hpx-1.10.0/libs/core/schedulers/include/hpx/schedulers/thread_queue.hpp:499

Direct leak of 1344 byte(s) in 21 object(s) allocated from:
    #0 0x7fd08a1ab3f5 in __interceptor_aligned_alloc ../../../../src/libsanitizer/asan/asan_malloc_linux.cc:183
    #1 0x7fd0897256ee in hpx::lockfree::queue<hpx::threads::policies::thread_queue<std::mutex, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_fifo>::task_description*, hpx::util::aligned_allocator<hpx::threads::policies::thread_queue<std::mutex, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_fifo>::task_description*, std::allocator<hpx::threads::policies::thread_queue<std::mutex, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_fifo>::task_description*> >, 0ul, false>::queue(unsigned long) /home/skillwon/Work/3rdParty/hpx-1.10.0/libs/core/allocator_support/include/hpx/allocator_support/aligned_allocator.hpp:50
    #2 0x7fd0897256ee in hpx::threads::policies::lockfree_fifo_backend<hpx::threads::policies::thread_queue<std::mutex, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_fifo>::task_description*>::lockfree_fifo_backend(unsigned long, unsigned long) /home/skillwon/Work/3rdParty/hpx-1.10.0/libs/core/schedulers/include/hpx/schedulers/lockfree_queue_backends.hpp:57
    #3 0x7fd0897256ee in hpx::threads::policies::thread_queue<std::mutex, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_fifo>::thread_queue(hpx::threads::policies::thread_queue_init_parameters const&) /home/skillwon/Work/3rdParty/hpx-1.10.0/libs/core/schedulers/include/hpx/schedulers/thread_queue.hpp:499

Direct leak of 192 byte(s) in 3 object(s) allocated from:
    #0 0x7fd08a1ab3f5 in __interceptor_aligned_alloc ../../../../src/libsanitizer/asan/asan_malloc_linux.cc:183
    #1 0x7fd089e6b74e in hpx::lockfree::queue<hpx::move_only_function<void (), false>*, hpx::util::aligned_allocator<hpx::move_only_function<void (), false>*, std::allocator<hpx::move_only_function<void (), false>*> >, 0ul, false>::queue(unsigned long) /home/skillwon/Work/3rdParty/hpx-1.10.0/libs/core/allocator_support/include/hpx/allocator_support/aligned_allocator.hpp:50
    #2 0x7fd089e6b74e in hpx::agas::big_boot_barrier::big_boot_barrier(hpx::parcelset::parcelport*, std::map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, hpx::parcelset::locality, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, hpx::parcelset::locality> > > const&, hpx::util::runtime_configuration const&) /home/skillwon/Work/3rdParty/hpx-1.10.0/libs/full/runtime_distributed/src/big_boot_barrier.cpp:686

Indirect leak of 105408 byte(s) in 1647 object(s) allocated from:
    #0 0x7fd08a1ab3f5 in __interceptor_aligned_alloc ../../../../src/libsanitizer/asan/asan_malloc_linux.cc:183
    #1 0x7fd0897256ee in hpx::lockfree::queue<hpx::threads::policies::thread_queue<std::mutex, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_fifo>::task_description*, hpx::util::aligned_allocator<hpx::threads::policies::thread_queue<std::mutex, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_fifo>::task_description*, std::allocator<hpx::threads::policies::thread_queue<std::mutex, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_fifo>::task_description*> >, 0ul, false>::queue(unsigned long) /home/skillwon/Work/3rdParty/hpx-1.10.0/libs/core/allocator_support/include/hpx/allocator_support/aligned_allocator.hpp:50
    #2 0x7fd0897256ee in hpx::threads::policies::lockfree_fifo_backend<hpx::threads::policies::thread_queue<std::mutex, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_fifo>::task_description*>::lockfree_fifo_backend(unsigned long, unsigned long) /home/skillwon/Work/3rdParty/hpx-1.10.0/libs/core/schedulers/include/hpx/schedulers/lockfree_queue_backends.hpp:57
    #3 0x7fd0897256ee in hpx::threads::policies::thread_queue<std::mutex, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_fifo>::thread_queue(hpx::threads::policies::thread_queue_init_parameters const&) /home/skillwon/Work/3rdParty/hpx-1.10.0/libs/core/schedulers/include/hpx/schedulers/thread_queue.hpp:499

Indirect leak of 104896 byte(s) in 1639 object(s) allocated from:
    #0 0x7fd08a1ab3f5 in __interceptor_aligned_alloc ../../../../src/libsanitizer/asan/asan_malloc_linux.cc:183
    #1 0x7fd0897254ae in hpx::lockfree::queue<hpx::threads::detail::thread_data_reference_counting*, hpx::util::aligned_allocator<hpx::threads::detail::thread_data_reference_counting*, std::allocator<hpx::threads::detail::thread_data_reference_counting*> >, 0ul, false>::queue(unsigned long) /home/skillwon/Work/3rdParty/hpx-1.10.0/libs/core/allocator_support/include/hpx/allocator_support/aligned_allocator.hpp:50
    #2 0x7fd0897254ae in hpx::threads::policies::lockfree_fifo_backend<hpx::threads::detail::thread_data_reference_counting*>::lockfree_fifo_backend(unsigned long, unsigned long) /home/skillwon/Work/3rdParty/hpx-1.10.0/libs/core/schedulers/include/hpx/schedulers/lockfree_queue_backends.hpp:57
    #3 0x7fd0897254ae in hpx::threads::policies::thread_queue<std::mutex, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_fifo>::thread_queue(hpx::threads::policies::thread_queue_init_parameters const&) /home/skillwon/Work/3rdParty/hpx-1.10.0/libs/core/schedulers/include/hpx/schedulers/thread_queue.hpp:499

Indirect leak of 79360 byte(s) in 1240 object(s) allocated from:
    #0 0x7fd08a1ab3f5 in __interceptor_aligned_alloc ../../../../src/libsanitizer/asan/asan_malloc_linux.cc:183
    #1 0x7fd0897255ce in hpx::lockfree::queue<hpx::threads::thread_data*, hpx::util::aligned_allocator<hpx::threads::thread_data*, std::allocator<hpx::threads::thread_data*> >, 0ul, false>::queue(unsigned long) /home/skillwon/Work/3rdParty/hpx-1.10.0/libs/core/allocator_support/include/hpx/allocator_support/aligned_allocator.hpp:50
    #2 0x7fd0897255ce in hpx::threads::policies::lockfree_fifo_backend<hpx::threads::thread_data*>::lockfree_fifo_backend(unsigned long, unsigned long) /home/skillwon/Work/3rdParty/hpx-1.10.0/libs/core/schedulers/include/hpx/schedulers/lockfree_queue_backends.hpp:57
    #3 0x7fd0897255ce in hpx::threads::policies::thread_queue<std::mutex, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_fifo>::thread_queue(hpx::threads::policies::thread_queue_init_parameters const&) /home/skillwon/Work/3rdParty/hpx-1.10.0/libs/core/schedulers/include/hpx/schedulers/thread_queue.hpp:499

Indirect leak of 1920 byte(s) in 30 object(s) allocated from:
    #0 0x7fd08a1ab3f5 in __interceptor_aligned_alloc ../../../../src/libsanitizer/asan/asan_malloc_linux.cc:183
    #1 0x7fd089e6b74e in hpx::lockfree::queue<hpx::move_only_function<void (), false>*, hpx::util::aligned_allocator<hpx::move_only_function<void (), false>*, std::allocator<hpx::move_only_function<void (), false>*> >, 0ul, false>::queue(unsigned long) /home/skillwon/Work/3rdParty/hpx-1.10.0/libs/core/allocator_support/include/hpx/allocator_support/aligned_allocator.hpp:50
    #2 0x7fd089e6b74e in hpx::agas::big_boot_barrier::big_boot_barrier(hpx::parcelset::parcelport*, std::map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, hpx::parcelset::locality, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, hpx::parcelset::locality> > > const&, hpx::util::runtime_configuration const&) /home/skillwon/Work/3rdParty/hpx-1.10.0/libs/full/runtime_distributed/src/big_boot_barrier.cpp:686

SUMMARY: AddressSanitizer: 322880 byte(s) leaked in 5045 allocation(s).

=================================================================
==864844==ERROR: LeakSanitizer: detected memory leaks

Direct leak of 27648 byte(s) in 432 object(s) allocated from:
    #0 0x7fd08a1ab3f5 in __interceptor_aligned_alloc ../../../../src/libsanitizer/asan/asan_malloc_linux.cc:183
    #1 0x7fd0897255ce in hpx::lockfree::queue<hpx::threads::thread_data*, hpx::util::aligned_allocator<hpx::threads::thread_data*, std::allocator<hpx::threads::thread_data*> >, 0ul, false>::queue(unsigned long) /home/skillwon/Work/3rdParty/hpx-1.10.0/libs/core/allocator_support/include/hpx/allocator_support/aligned_allocator.hpp:50
    #2 0x7fd0897255ce in hpx::threads::policies::lockfree_fifo_backend<hpx::threads::thread_data*>::lockfree_fifo_backend(unsigned long, unsigned long) /home/skillwon/Work/3rdParty/hpx-1.10.0/libs/core/schedulers/include/hpx/schedulers/lockfree_queue_backends.hpp:57
    #3 0x7fd0897255ce in hpx::threads::policies::thread_queue<std::mutex, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_fifo>::thread_queue(hpx::threads::policies::thread_queue_init_parameters const&) /home/skillwon/Work/3rdParty/hpx-1.10.0/libs/core/schedulers/include/hpx/schedulers/thread_queue.hpp:499

Direct leak of 2112 byte(s) in 33 object(s) allocated from:
    #0 0x7fd08a1ab3f5 in __interceptor_aligned_alloc ../../../../src/libsanitizer/asan/asan_malloc_linux.cc:183
    #1 0x7fd0897254ae in hpx::lockfree::queue<hpx::threads::detail::thread_data_reference_counting*, hpx::util::aligned_allocator<hpx::threads::detail::thread_data_reference_counting*, std::allocator<hpx::threads::detail::thread_data_reference_counting*> >, 0ul, false>::queue(unsigned long) /home/skillwon/Work/3rdParty/hpx-1.10.0/libs/core/allocator_support/include/hpx/allocator_support/aligned_allocator.hpp:50
    #2 0x7fd0897254ae in hpx::threads::policies::lockfree_fifo_backend<hpx::threads::detail::thread_data_reference_counting*>::lockfree_fifo_backend(unsigned long, unsigned long) /home/skillwon/Work/3rdParty/hpx-1.10.0/libs/core/schedulers/include/hpx/schedulers/lockfree_queue_backends.hpp:57
    #3 0x7fd0897254ae in hpx::threads::policies::thread_queue<std::mutex, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_fifo>::thread_queue(hpx::threads::policies::thread_queue_init_parameters const&) /home/skillwon/Work/3rdParty/hpx-1.10.0/libs/core/schedulers/include/hpx/schedulers/thread_queue.hpp:499

Direct leak of 1344 byte(s) in 21 object(s) allocated from:
    #0 0x7fd08a1ab3f5 in __interceptor_aligned_alloc ../../../../src/libsanitizer/asan/asan_malloc_linux.cc:183
    #1 0x7fd0897256ee in hpx::lockfree::queue<hpx::threads::policies::thread_queue<std::mutex, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_fifo>::task_description*, hpx::util::aligned_allocator<hpx::threads::policies::thread_queue<std::mutex, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_fifo>::task_description*, std::allocator<hpx::threads::policies::thread_queue<std::mutex, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_fifo>::task_description*> >, 0ul, false>::queue(unsigned long) /home/skillwon/Work/3rdParty/hpx-1.10.0/libs/core/allocator_support/include/hpx/allocator_support/aligned_allocator.hpp:50
    #2 0x7fd0897256ee in hpx::threads::policies::lockfree_fifo_backend<hpx::threads::policies::thread_queue<std::mutex, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_fifo>::task_description*>::lockfree_fifo_backend(unsigned long, unsigned long) /home/skillwon/Work/3rdParty/hpx-1.10.0/libs/core/schedulers/include/hpx/schedulers/lockfree_queue_backends.hpp:57
    #3 0x7fd0897256ee in hpx::threads::policies::thread_queue<std::mutex, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_fifo>::thread_queue(hpx::threads::policies::thread_queue_init_parameters const&) /home/skillwon/Work/3rdParty/hpx-1.10.0/libs/core/schedulers/include/hpx/schedulers/thread_queue.hpp:499

Direct leak of 192 byte(s) in 3 object(s) allocated from:
    #0 0x7fd08a1ab3f5 in __interceptor_aligned_alloc ../../../../src/libsanitizer/asan/asan_malloc_linux.cc:183
    #1 0x7fd089e6b74e in hpx::lockfree::queue<hpx::move_only_function<void (), false>*, hpx::util::aligned_allocator<hpx::move_only_function<void (), false>*, std::allocator<hpx::move_only_function<void (), false>*> >, 0ul, false>::queue(unsigned long) /home/skillwon/Work/3rdParty/hpx-1.10.0/libs/core/allocator_support/include/hpx/allocator_support/aligned_allocator.hpp:50
    #2 0x7fd089e6b74e in hpx::agas::big_boot_barrier::big_boot_barrier(hpx::parcelset::parcelport*, std::map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, hpx::parcelset::locality, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, hpx::parcelset::locality> > > const&, hpx::util::runtime_configuration const&) /home/skillwon/Work/3rdParty/hpx-1.10.0/libs/full/runtime_distributed/src/big_boot_barrier.cpp:686

Indirect leak of 105408 byte(s) in 1647 object(s) allocated from:
    #0 0x7fd08a1ab3f5 in __interceptor_aligned_alloc ../../../../src/libsanitizer/asan/asan_malloc_linux.cc:183
    #1 0x7fd0897256ee in hpx::lockfree::queue<hpx::threads::policies::thread_queue<std::mutex, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_fifo>::task_description*, hpx::util::aligned_allocator<hpx::threads::policies::thread_queue<std::mutex, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_fifo>::task_description*, std::allocator<hpx::threads::policies::thread_queue<std::mutex, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_fifo>::task_description*> >, 0ul, false>::queue(unsigned long) /home/skillwon/Work/3rdParty/hpx-1.10.0/libs/core/allocator_support/include/hpx/allocator_support/aligned_allocator.hpp:50
    #2 0x7fd0897256ee in hpx::threads::policies::lockfree_fifo_backend<hpx::threads::policies::thread_queue<std::mutex, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_fifo>::task_description*>::lockfree_fifo_backend(unsigned long, unsigned long) /home/skillwon/Work/3rdParty/hpx-1.10.0/libs/core/schedulers/include/hpx/schedulers/lockfree_queue_backends.hpp:57
    #3 0x7fd0897256ee in hpx::threads::policies::thread_queue<std::mutex, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_fifo>::thread_queue(hpx::threads::policies::thread_queue_init_parameters const&) /home/skillwon/Work/3rdParty/hpx-1.10.0/libs/core/schedulers/include/hpx/schedulers/thread_queue.hpp:499

Indirect leak of 104896 byte(s) in 1639 object(s) allocated from:
    #0 0x7fd08a1ab3f5 in __interceptor_aligned_alloc ../../../../src/libsanitizer/asan/asan_malloc_linux.cc:183
    #1 0x7fd0897254ae in hpx::lockfree::queue<hpx::threads::detail::thread_data_reference_counting*, hpx::util::aligned_allocator<hpx::threads::detail::thread_data_reference_counting*, std::allocator<hpx::threads::detail::thread_data_reference_counting*> >, 0ul, false>::queue(unsigned long) /home/skillwon/Work/3rdParty/hpx-1.10.0/libs/core/allocator_support/include/hpx/allocator_support/aligned_allocator.hpp:50
    #2 0x7fd0897254ae in hpx::threads::policies::lockfree_fifo_backend<hpx::threads::detail::thread_data_reference_counting*>::lockfree_fifo_backend(unsigned long, unsigned long) /home/skillwon/Work/3rdParty/hpx-1.10.0/libs/core/schedulers/include/hpx/schedulers/lockfree_queue_backends.hpp:57
    #3 0x7fd0897254ae in hpx::threads::policies::thread_queue<std::mutex, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_fifo>::thread_queue(hpx::threads::policies::thread_queue_init_parameters const&) /home/skillwon/Work/3rdParty/hpx-1.10.0/libs/core/schedulers/include/hpx/schedulers/thread_queue.hpp:499

Indirect leak of 79360 byte(s) in 1240 object(s) allocated from:
    #0 0x7fd08a1ab3f5 in __interceptor_aligned_alloc ../../../../src/libsanitizer/asan/asan_malloc_linux.cc:183
    #1 0x7fd0897255ce in hpx::lockfree::queue<hpx::threads::thread_data*, hpx::util::aligned_allocator<hpx::threads::thread_data*, std::allocator<hpx::threads::thread_data*> >, 0ul, false>::queue(unsigned long) /home/skillwon/Work/3rdParty/hpx-1.10.0/libs/core/allocator_support/include/hpx/allocator_support/aligned_allocator.hpp:50
    #2 0x7fd0897255ce in hpx::threads::policies::lockfree_fifo_backend<hpx::threads::thread_data*>::lockfree_fifo_backend(unsigned long, unsigned long) /home/skillwon/Work/3rdParty/hpx-1.10.0/libs/core/schedulers/include/hpx/schedulers/lockfree_queue_backends.hpp:57
    #3 0x7fd0897255ce in hpx::threads::policies::thread_queue<std::mutex, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_fifo>::thread_queue(hpx::threads::policies::thread_queue_init_parameters const&) /home/skillwon/Work/3rdParty/hpx-1.10.0/libs/core/schedulers/include/hpx/schedulers/thread_queue.hpp:499

Indirect leak of 1920 byte(s) in 30 object(s) allocated from:
    #0 0x7fd08a1ab3f5 in __interceptor_aligned_alloc ../../../../src/libsanitizer/asan/asan_malloc_linux.cc:183
    #1 0x7fd089e6b74e in hpx::lockfree::queue<hpx::move_only_function<void (), false>*, hpx::util::aligned_allocator<hpx::move_only_function<void (), false>*, std::allocator<hpx::move_only_function<void (), false>*> >, 0ul, false>::queue(unsigned long) /home/skillwon/Work/3rdParty/hpx-1.10.0/libs/core/allocator_support/include/hpx/allocator_support/aligned_allocator.hpp:50
    #2 0x7fd089e6b74e in hpx::agas::big_boot_barrier::big_boot_barrier(hpx::parcelset::parcelport*, std::map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, hpx::parcelset::locality, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, hpx::parcelset::locality> > > const&, hpx::util::runtime_configuration const&) /home/skillwon/Work/3rdParty/hpx-1.10.0/libs/full/runtime_distributed/src/big_boot_barrier.cpp:686

SUMMARY: AddressSanitizer: 322880 byte(s) leaked in 5045 allocation(s).

=================================================================
==864844==ERROR: LeakSanitizer: detected memory leaks

Direct leak of 27648 byte(s) in 432 object(s) allocated from:
    #0 0x7fd08a1ab3f5 in __interceptor_aligned_alloc ../../../../src/libsanitizer/asan/asan_malloc_linux.cc:183
    #1 0x7fd0897255ce in hpx::lockfree::queue<hpx::threads::thread_data*, hpx::util::aligned_allocator<hpx::threads::thread_data*, std::allocator<hpx::threads::thread_data*> >, 0ul, false>::queue(unsigned long) /home/skillwon/Work/3rdParty/hpx-1.10.0/libs/core/allocator_support/include/hpx/allocator_support/aligned_allocator.hpp:50
    #2 0x7fd0897255ce in hpx::threads::policies::lockfree_fifo_backend<hpx::threads::thread_data*>::lockfree_fifo_backend(unsigned long, unsigned long) /home/skillwon/Work/3rdParty/hpx-1.10.0/libs/core/schedulers/include/hpx/schedulers/lockfree_queue_backends.hpp:57
    #3 0x7fd0897255ce in hpx::threads::policies::thread_queue<std::mutex, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_fifo>::thread_queue(hpx::threads::policies::thread_queue_init_parameters const&) /home/skillwon/Work/3rdParty/hpx-1.10.0/libs/core/schedulers/include/hpx/schedulers/thread_queue.hpp:499

Direct leak of 2112 byte(s) in 33 object(s) allocated from:
    #0 0x7fd08a1ab3f5 in __interceptor_aligned_alloc ../../../../src/libsanitizer/asan/asan_malloc_linux.cc:183
    #1 0x7fd0897254ae in hpx::lockfree::queue<hpx::threads::detail::thread_data_reference_counting*, hpx::util::aligned_allocator<hpx::threads::detail::thread_data_reference_counting*, std::allocator<hpx::threads::detail::thread_data_reference_counting*> >, 0ul, false>::queue(unsigned long) /home/skillwon/Work/3rdParty/hpx-1.10.0/libs/core/allocator_support/include/hpx/allocator_support/aligned_allocator.hpp:50
    #2 0x7fd0897254ae in hpx::threads::policies::lockfree_fifo_backend<hpx::threads::detail::thread_data_reference_counting*>::lockfree_fifo_backend(unsigned long, unsigned long) /home/skillwon/Work/3rdParty/hpx-1.10.0/libs/core/schedulers/include/hpx/schedulers/lockfree_queue_backends.hpp:57
    #3 0x7fd0897254ae in hpx::threads::policies::thread_queue<std::mutex, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_fifo>::thread_queue(hpx::threads::policies::thread_queue_init_parameters const&) /home/skillwon/Work/3rdParty/hpx-1.10.0/libs/core/schedulers/include/hpx/schedulers/thread_queue.hpp:499

Direct leak of 1344 byte(s) in 21 object(s) allocated from:
    #0 0x7fd08a1ab3f5 in __interceptor_aligned_alloc ../../../../src/libsanitizer/asan/asan_malloc_linux.cc:183
    #1 0x7fd0897256ee in hpx::lockfree::queue<hpx::threads::policies::thread_queue<std::mutex, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_fifo>::task_description*, hpx::util::aligned_allocator<hpx::threads::policies::thread_queue<std::mutex, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_fifo>::task_description*, std::allocator<hpx::threads::policies::thread_queue<std::mutex, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_fifo>::task_description*> >, 0ul, false>::queue(unsigned long) /home/skillwon/Work/3rdParty/hpx-1.10.0/libs/core/allocator_support/include/hpx/allocator_support/aligned_allocator.hpp:50
    #2 0x7fd0897256ee in hpx::threads::policies::lockfree_fifo_backend<hpx::threads::policies::thread_queue<std::mutex, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_fifo>::task_description*>::lockfree_fifo_backend(unsigned long, unsigned long) /home/skillwon/Work/3rdParty/hpx-1.10.0/libs/core/schedulers/include/hpx/schedulers/lockfree_queue_backends.hpp:57
    #3 0x7fd0897256ee in hpx::threads::policies::thread_queue<std::mutex, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_fifo>::thread_queue(hpx::threads::policies::thread_queue_init_parameters const&) /home/skillwon/Work/3rdParty/hpx-1.10.0/libs/core/schedulers/include/hpx/schedulers/thread_queue.hpp:499

Direct leak of 192 byte(s) in 3 object(s) allocated from:
    #0 0x7fd08a1ab3f5 in __interceptor_aligned_alloc ../../../../src/libsanitizer/asan/asan_malloc_linux.cc:183
    #1 0x7fd089e6b74e in hpx::lockfree::queue<hpx::move_only_function<void (), false>*, hpx::util::aligned_allocator<hpx::move_only_function<void (), false>*, std::allocator<hpx::move_only_function<void (), false>*> >, 0ul, false>::queue(unsigned long) /home/skillwon/Work/3rdParty/hpx-1.10.0/libs/core/allocator_support/include/hpx/allocator_support/aligned_allocator.hpp:50
    #2 0x7fd089e6b74e in hpx::agas::big_boot_barrier::big_boot_barrier(hpx::parcelset::parcelport*, std::map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, hpx::parcelset::locality, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, hpx::parcelset::locality> > > const&, hpx::util::runtime_configuration const&) /home/skillwon/Work/3rdParty/hpx-1.10.0/libs/full/runtime_distributed/src/big_boot_barrier.cpp:686

Indirect leak of 105408 byte(s) in 1647 object(s) allocated from:
    #0 0x7fd08a1ab3f5 in __interceptor_aligned_alloc ../../../../src/libsanitizer/asan/asan_malloc_linux.cc:183
    #1 0x7fd0897256ee in hpx::lockfree::queue<hpx::threads::policies::thread_queue<std::mutex, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_fifo>::task_description*, hpx::util::aligned_allocator<hpx::threads::policies::thread_queue<std::mutex, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_fifo>::task_description*, std::allocator<hpx::threads::policies::thread_queue<std::mutex, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_fifo>::task_description*> >, 0ul, false>::queue(unsigned long) /home/skillwon/Work/3rdParty/hpx-1.10.0/libs/core/allocator_support/include/hpx/allocator_support/aligned_allocator.hpp:50
    #2 0x7fd0897256ee in hpx::threads::policies::lockfree_fifo_backend<hpx::threads::policies::thread_queue<std::mutex, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_fifo>::task_description*>::lockfree_fifo_backend(unsigned long, unsigned long) /home/skillwon/Work/3rdParty/hpx-1.10.0/libs/core/schedulers/include/hpx/schedulers/lockfree_queue_backends.hpp:57
    #3 0x7fd0897256ee in hpx::threads::policies::thread_queue<std::mutex, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_fifo>::thread_queue(hpx::threads::policies::thread_queue_init_parameters const&) /home/skillwon/Work/3rdParty/hpx-1.10.0/libs/core/schedulers/include/hpx/schedulers/thread_queue.hpp:499

Indirect leak of 104896 byte(s) in 1639 object(s) allocated from:
    #0 0x7fd08a1ab3f5 in __interceptor_aligned_alloc ../../../../src/libsanitizer/asan/asan_malloc_linux.cc:183
    #1 0x7fd0897254ae in hpx::lockfree::queue<hpx::threads::detail::thread_data_reference_counting*, hpx::util::aligned_allocator<hpx::threads::detail::thread_data_reference_counting*, std::allocator<hpx::threads::detail::thread_data_reference_counting*> >, 0ul, false>::queue(unsigned long) /home/skillwon/Work/3rdParty/hpx-1.10.0/libs/core/allocator_support/include/hpx/allocator_support/aligned_allocator.hpp:50
    #2 0x7fd0897254ae in hpx::threads::policies::lockfree_fifo_backend<hpx::threads::detail::thread_data_reference_counting*>::lockfree_fifo_backend(unsigned long, unsigned long) /home/skillwon/Work/3rdParty/hpx-1.10.0/libs/core/schedulers/include/hpx/schedulers/lockfree_queue_backends.hpp:57
    #3 0x7fd0897254ae in hpx::threads::policies::thread_queue<std::mutex, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_fifo>::thread_queue(hpx::threads::policies::thread_queue_init_parameters const&) /home/skillwon/Work/3rdParty/hpx-1.10.0/libs/core/schedulers/include/hpx/schedulers/thread_queue.hpp:499

Indirect leak of 79360 byte(s) in 1240 object(s) allocated from:
    #0 0x7fd08a1ab3f5 in __interceptor_aligned_alloc ../../../../src/libsanitizer/asan/asan_malloc_linux.cc:183
    #1 0x7fd0897255ce in hpx::lockfree::queue<hpx::threads::thread_data*, hpx::util::aligned_allocator<hpx::threads::thread_data*, std::allocator<hpx::threads::thread_data*> >, 0ul, false>::queue(unsigned long) /home/skillwon/Work/3rdParty/hpx-1.10.0/libs/core/allocator_support/include/hpx/allocator_support/aligned_allocator.hpp:50
    #2 0x7fd0897255ce in hpx::threads::policies::lockfree_fifo_backend<hpx::threads::thread_data*>::lockfree_fifo_backend(unsigned long, unsigned long) /home/skillwon/Work/3rdParty/hpx-1.10.0/libs/core/schedulers/include/hpx/schedulers/lockfree_queue_backends.hpp:57
    #3 0x7fd0897255ce in hpx::threads::policies::thread_queue<std::mutex, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_fifo, hpx::threads::policies::lockfree_fifo>::thread_queue(hpx::threads::policies::thread_queue_init_parameters const&) /home/skillwon/Work/3rdParty/hpx-1.10.0/libs/core/schedulers/include/hpx/schedulers/thread_queue.hpp:499

Indirect leak of 1920 byte(s) in 30 object(s) allocated from:
    #0 0x7fd08a1ab3f5 in __interceptor_aligned_alloc ../../../../src/libsanitizer/asan/asan_malloc_linux.cc:183
    #1 0x7fd089e6b74e in hpx::lockfree::queue<hpx::move_only_function<void (), false>*, hpx::util::aligned_allocator<hpx::move_only_function<void (), false>*, std::allocator<hpx::move_only_function<void (), false>*> >, 0ul, false>::queue(unsigned long) /home/skillwon/Work/3rdParty/hpx-1.10.0/libs/core/allocator_support/include/hpx/allocator_support/aligned_allocator.hpp:50
    #2 0x7fd089e6b74e in hpx::agas::big_boot_barrier::big_boot_barrier(hpx::parcelset::parcelport*, std::map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, hpx::parcelset::locality, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, hpx::parcelset::locality> > > const&, hpx::util::runtime_configuration const&) /home/skillwon/Work/3rdParty/hpx-1.10.0/libs/full/runtime_distributed/src/big_boot_barrier.cpp:686

SUMMARY: AddressSanitizer: 322880 byte(s) leaked in 5045 allocation(s).
...
...
...

I am not sure if this information is valid; in other words, I am uncertain whether HPX has actually experienced a memory leak. Could you please take a look? Thank you.

@hkaiser
Copy link
Member

hkaiser commented Nov 19, 2024

That's an interesting observation! The 'leaks' reported here are caused by the free-store associated with our lock-free queues. Those (currently) never return the memory they allocate, instead they put the memory back into a free-store (a special allocator) for later use.

The free-store is (supposed to be) free'd at the end of execution. Your report hints at that this is not the case. BTW, the fact that the other localities (besides root) don't produce any sanitizer output hints at the fact that the leak happens on the root locality only (which is consistent with your initial report as well).

Thanks for this information. I will have a closer look as soon as I can find the time. Would you be able to post the full ASAN output somewhere for me to download?

@phil-skillwon
Copy link
Author

@hkaiser
I have uploaded the ASAN output file as an attachment.
As I mentioned in my previous response, I recompiled HPX and added some compilation options to my test code (CMake project) to support ASAN. However, I'm not sure whether this would affect ASAN's ability to correctly diagnose memory leaks.
Honestly, I'm not certain whether this information is a false positive from ASAN.
Based on your previous response, can we confirm that a memory leak occurred on the root node?

By the way, in my test code, I created a thread that calls the __lsan_do_recoverable_leak_check function every 60 seconds to output ASAN debug information.
All ASAN debug information is output through the __lsan_do_recoverable_leak_check function in this thread.
Here is the code:

#include <sanitizer/lsan_interface.h>

int hpx_main(int argc, char* argv[])
{
	...
	...
	...

	auto asan = hpx::thread([&](){
        while (true) 
        {
            hpx::this_thread::sleep_for(60s);
            __lsan_do_recoverable_leak_check();
        }

    });
	
	...
	...
	...
	if (true == asan.joinable()) 
    {
        asan.join();
    }
    else 
    {
        asan.detach();
    }
	
	return hpx::finalize();
}

Here is the description of these two functions from the sanitizer:

sanitizer/lsan_interface.h

// Check for leaks now. This function behaves identically to the default
// end-of-process leak check. In particular, it will terminate the process if
// leaks are found and the exitcode runtime flag is non-zero.
// Subsequent calls to this function will have no effect and end-of-process
// leak check will not run. Effectively, end-of-process leak check is moved to
// the time of first invocation of this function.
// By calling this function early during process shutdown, you can instruct
// LSan to ignore shutdown-only leaks which happen later on.
void __lsan_do_leak_check(void);

// Check for leaks now. Returns zero if no leaks have been found or if leak
// detection is disabled, non-zero otherwise.
// This function may be called repeatedly, e.g. to periodically check a
// long-running process. It prints a leak report if appropriate, but does not
// terminate the process. It does not affect the behavior of
// __lsan_do_leak_check() or the end-of-process leak check, and is not
// affected by them.
int __lsan_do_recoverable_leak_check(void);

Asan.log.zip

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
category: parcel transport type: defect
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@hkaiser @phil-skillwon