Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

can not trace log_func_args when import torch #338

Open
yihong0618 opened this issue May 16, 2023 · 3 comments
Open

can not trace log_func_args when import torch #338

yihong0618 opened this issue May 16, 2023 · 3 comments
Labels
need further investigation Don't need fix right away, but worth looking at

Comments

@yihong0618
Copy link
Contributor

yihong0618 commented May 16, 2023

os: ubuntu20.04 and MacOS m1 both test

cat foo.py

import torch

def bar():
    return 3


bar()

run this command the viztracer will hang

viztracer --log_func_args   a.py

use py-spy to check the stack will see as follows:

image

seems loop in getattr (torch/_ops.py:480)

....
....
....
    __getattr__ (torch/_ops.py:480)
    __getattr__ (torch/_ops.py:480)
    __getattr__ (torch/_ops.py:480)
    __getattr__ (torch/_ops.py:480)
    __getattr__ (torch/_ops.py:480)
    __getattr__ (torch/_ops.py:480)
    __getattr__ (torch/_ops.py:480)
    __getattr__ (torch/_ops.py:480)
    __getattr__ (torch/_ops.py:480)
    __getattr__ (torch/_ops.py:480)
    __getattr__ (torch/_ops.py:480)
    __getattr__ (torch/_ops.py:480)
    __getattr__ (torch/_ops.py:480)
    __getattr__ (torch/_ops.py:480)
    __getattr__ (torch/_ops.py:480)
    __getattr__ (torch/_ops.py:480)
    __getattr__ (torch/_ops.py:480)
    __getattr__ (torch/_ops.py:480)
    __getattr__ (torch/_ops.py:480)
    __getattr__ (torch/_ops.py:480)
    __getattr__ (torch/_ops.py:480)
    __getattr__ (torch/_ops.py:480)
    __repr__ (torch/_ops.py:434)
    __init__ (torch/_ops.py:422)
    __getattr__ (torch/_ops.py:581)
    <module> (torch/fx/node.py:35)
    _call_with_frames_removed (<frozen importlib._bootstrap>:241)
    exec_module (<frozen importlib._bootstrap_external>:883)
    _load_unlocked (<frozen importlib._bootstrap>:688)
    _find_and_load_unlocked (<frozen importlib._bootstrap>:1006)
    _find_and_load (<frozen importlib._bootstrap>:1027)
    <module> (torch/fx/graph.py:2)
    _call_with_frames_removed (<frozen importlib._bootstrap>:241)
    exec_module (<frozen importlib._bootstrap_external>:883)
    _load_unlocked (<frozen importlib._bootstrap>:688)
    _find_and_load_unlocked (<frozen importlib._bootstrap>:1006)
    _find_and_load (<frozen importlib._bootstrap>:1027)
    <module> (torch/fx/graph_module.py:8)
    _call_with_frames_removed (<frozen importlib._bootstrap>:241)
    exec_module (<frozen importlib._bootstrap_external>:883)
    _load_unlocked (<frozen importlib._bootstrap>:688)
    _find_and_load_unlocked (<frozen importlib._bootstrap>:1006)
    _find_and_load (<frozen importlib._bootstrap>:1027)
    <module> (torch/fx/__init__.py:83)
    _call_with_frames_removed (<frozen importlib._bootstrap>:241)
    exec_module (<frozen importlib._bootstrap_external>:883)
    _load_unlocked (<frozen importlib._bootstrap>:688)
    _find_and_load_unlocked (<frozen importlib._bootstrap>:1006)
    _find_and_load (<frozen importlib._bootstrap>:1027)
    <module> (torch/ao/quantization/utils.py:12)
    _call_with_frames_removed (<frozen importlib._bootstrap>:241)
    exec_module (<frozen importlib._bootstrap_external>:883)
    _load_unlocked (<frozen importlib._bootstrap>:688)
    _find_and_load_unlocked (<frozen importlib._bootstrap>:1006)
    _find_and_load (<frozen importlib._bootstrap>:1027)
    <module> (torch/ao/quantization/observer.py:15)
    _call_with_frames_removed (<frozen importlib._bootstrap>:241)
    exec_module (<frozen importlib._bootstrap_external>:883)
    _load_unlocked (<frozen importlib._bootstrap>:688)
    _find_and_load_unlocked (<frozen importlib._bootstrap>:1006)
    _find_and_load (<frozen importlib._bootstrap>:1027)
    <module> (torch/ao/quantization/fake_quantize.py:8)
    _call_with_frames_removed (<frozen importlib._bootstrap>:241)
    exec_module (<frozen importlib._bootstrap_external>:883)
    _load_unlocked (<frozen importlib._bootstrap>:688)
    _find_and_load_unlocked (<frozen importlib._bootstrap>:1006)
    _find_and_load (<frozen importlib._bootstrap>:1027)
    <module> (torch/ao/quantization/__init__.py:3)
    _call_with_frames_removed (<frozen importlib._bootstrap>:241)
    exec_module (<frozen importlib._bootstrap_external>:883)
    _load_unlocked (<frozen importlib._bootstrap>:688)
    _find_and_load_unlocked (<frozen importlib._bootstrap>:1006)
    _find_and_load (<frozen importlib._bootstrap>:1027)
    _call_with_frames_removed (<frozen importlib._bootstrap>:241)
    _find_and_load_unlocked (<frozen importlib._bootstrap>:992)
    _find_and_load (<frozen importlib._bootstrap>:1027)
    _find_and_load_unlocked (<frozen importlib._bootstrap>:992)
    _find_and_load (<frozen importlib._bootstrap>:1027)
    <module> (torch/quantization/quantize.py:10)
    _call_with_frames_removed (<frozen importlib._bootstrap>:241)
    exec_module (<frozen importlib._bootstrap_external>:883)
    _load_unlocked (<frozen importlib._bootstrap>:688)
    _find_and_load_unlocked (<frozen importlib._bootstrap>:1006)
    _find_and_load (<frozen importlib._bootstrap>:1027)
    <module> (torch/quantization/__init__.py:1)
    _call_with_frames_removed (<frozen importlib._bootstrap>:241)
    exec_module (<frozen importlib._bootstrap_external>:883)
    _load_unlocked (<frozen importlib._bootstrap>:688)
    _find_and_load_unlocked (<frozen importlib._bootstrap>:1006)
    _find_and_load (<frozen importlib._bootstrap>:1027)
    _call_with_frames_removed (<frozen importlib._bootstrap>:241)
    _handle_fromlist (<frozen importlib._bootstrap>:1078)
    <module> (torch/__init__.py:1288)
    _call_with_frames_removed (<frozen importlib._bootstrap>:241)
    exec_module (<frozen importlib._bootstrap_external>:883)
    _load_unlocked (<frozen importlib._bootstrap>:688)
    _find_and_load_unlocked (<frozen importlib._bootstrap>:1006)
    _find_and_load (<frozen importlib._bootstrap>:1027)
    <module> (a.py:2)
    run_code (viztracer/main.py:334)
    run_command (viztracer/main.py:417)
    run (viztracer/main.py:301)
    main (viztracer/main.py:566)
    <module> (viztracer:8)
@yihong0618 yihong0618 changed the title can not trace log_func_args when import torce can not trace log_func_args when import torch May 16, 2023
@yihong0618
Copy link
Contributor Author

after some search the root cause seems from PyObject_Repr(PyDict_GetItem(locals, name));
the repr and getattar caused the recursive.
but I am not sure it needs fix on viztracer or pytorch side.

@gaogaotiantian

@gaogaotiantian
Copy link
Owner

It seems like using getattr on that object will cause infinite recursion? If this could be reproduced without viztracer, maybe it’s not a viztracer issue. Is there a note in torch for this? That users should not do it?

@yihong0618
Copy link
Contributor Author

not exactly I did some search and no relate issues on pytorch.
but in cpython I found this one python/cpython#86075 similar but not the same thing.
I will try use bare sys.settrace to test later

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
need further investigation Don't need fix right away, but worth looking at
Projects
None yet
Development

No branches or pull requests

2 participants