Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Possible bug with dogstatsd deadlock #834

Open
mazad01 opened this issue Jun 14, 2024 · 4 comments
Open

Possible bug with dogstatsd deadlock #834

mazad01 opened this issue Jun 14, 2024 · 4 comments
Labels
kind/bug Bug related issue stale Stale - Bot reminder

Comments

@mazad01
Copy link

mazad01 commented Jun 14, 2024

Describe the bug
Code is hanging on datadog's post_fork() function. Possibly https://github.com/DataDog/datadogpy/blob/master/datadog/dogstatsd/base.py#L976 ?

Expected behavior
Dogstatsd lib references not deadlocking

Environment and Versions (please complete the following information):
dogstatsd 0.49.1

Additional context
From our developers: the parent code is trying to spawn subprocesses, but it ultimately hangs

i.e snip of the code:

 process = subprocess.Popen(
        cmd,
        stdout=subprocess.PIPE,
        stderr=subprocess.STDOUT,
        shell=True,
        text=True,
        preexec_fn=_set_pdeathsig(
            signal.SIGTERM
        ),  # ensure terminate subproc if this proc dies
    )


def _set_pdeathsig(sig=signal.SIGTERM):
    """Return a callable that terminates the subprocess."""
    libc = ctypes.CDLL("libc.so.6")

    def callable():
        return libc.prctl(1, sig)

    return callable

Output of pyspy:

Thread 1733153 (idle): "Thread-52 (process_items)"
    close_socket (datadog/dogstatsd/base.py:933)
    post_fork (datadog/dogstatsd/base.py:1369)
    post_fork (datadog/dogstatsd/base.py:120)
    _execute_child (subprocess.py:1796)
    __init__ (subprocess.py:971)
...

Output of strace

strace: Process 1733153 attached
futex(0x5644f988f040, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 0, NULL, FUTEX_BITSET_MATCH_ANY
@mazad01 mazad01 added the kind/bug Bug related issue label Jun 14, 2024
@vickenty
Copy link
Contributor

Thank you for reporting the issue.

While we are working on a fix. In the meantime, you can work around the issue by disabling fork support by setting the DD_DOGSTATSD_DISABLE_FORK_SUPPORT environment variable before importing the datadog package.

Copy link

Thanks for your contribution!

This issue has been automatically marked as stale because it has not had activity in the last 30 days. Note that the issue will not be automatically closed, but this notification will remind us to investigate why there's been inactivity. Thank you for participating in the Datadog open source community.

If you would like this issue to remain open:

  1. Verify that you can still reproduce the issue in the latest version of this project.

  2. Comment that the issue is still reproducible and include updated details requested in the issue template.

@github-actions github-actions bot added the stale Stale - Bot reminder label Jul 22, 2024
@mazad01
Copy link
Author

mazad01 commented Jul 22, 2024

Not stale, still waiting for resolution. Thanks

@anupamdialpad
Copy link

What are the side effects of setting DD_DOGSTATSD_DISABLE_FORK_SUPPORT?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
kind/bug Bug related issue stale Stale - Bot reminder
Projects
None yet
Development

No branches or pull requests

3 participants