-
-
Notifications
You must be signed in to change notification settings - Fork 379
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Very long process of getting repr of huge objects in torch #461
Comments
VizTracer is not able to limit |
I didn't understand the part about repr in C, where it says that viztracer will not be triggered (invoked?). It's viztracer that calls repr, isn't it? I think it is possible to implement this using execution isolation (running repr in a separate thread, process, or asynchronous task to limit execution with a timeout). However, it sounds like a lot of effort just for handling torch, which can be simply filtered out through exclude files. Another way I can suppose - use sys.getsizeof() before calling repr and skip large files. Upd: Another one - print warning about unusual long repr during processing <file.py> to show user file which he might want to exclude |
Yes it's viztracer that calls Overall, if an object decides to make its |
The time of repr depends on size of object if the object repr implementation is supposed to iterate all of its stuff as in case of torch storage. However, I agree, that's a corner case, and we can't say wheather time of repr depends on size or not before invoking repr. What about running repr in separate executor, why will there be racing issue or stuck? And why we don't see it now? Can't we find a way of implementation without introducing new issues? |
Concurrency is not the silver bullet for everything that's slow. VizTracer needs the result of Is it possible to just move forward without the result? Maybe. But first of all - it's just not worth it because it's not worth it. You'll need to change a lot of the current structure to make it work. Then, what if the object changed during the process? That's the racing I talked about. So, overall, it's infeasible and not worth it to do it in a separate executor. Also, we will not do anything special just for |
okay, sounds reasonable, thank you for your time) |
viztracer --tracer_entries 10000000 --ignore_frozen --ignore_c_function --log_func_args --max_stack_depth 30 -- scripts\train_ui.py
it's not infinite recursion like in #338, just very long process. Maybe additional flag limiting repr process would solve both issues.
Max Stack depth option helps if it prevents from entrying into this objects, but if we are already in, it will not help us to get out
The text was updated successfully, but these errors were encountered: