Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Gives warning: "Found and ignore invalid json file, you may lost some process data." #436

Closed
adhadseKavida opened this issue May 14, 2024 · 6 comments

Comments

@adhadseKavida
Copy link

adhadseKavida commented May 14, 2024

Additionally prints this:.

Invalid json file:
    /tmp/tmp9thv7lj2/result_24189.json
    /tmp/tmp9thv7lj2/result_24336.json
    /tmp/tmp9thv7lj2/result_24337.json
    /tmp/tmp9thv7lj2/result_24187.json
    /tmp/tmp9thv7lj2/result_24168.json

What should be solution to this, trying to visualize the final generated files gives incomplete information.

@gaogaotiantian
Copy link
Owner

Did you use Windows + multiprocessing pools?

@adhadseKavida
Copy link
Author

adhadseKavida commented May 20, 2024

I'm on Linux, Fedora, yes. It's Multiprocessing + Async code in each process.

@gaogaotiantian
Copy link
Owner

Sorry I forgot to reply to this. The most common reason that the temp json file is not valid is because the process is shut down abruptly. For example, if there's any process.kill(). There's no way for viztracer to log the files if the process is just killed. However, I don't know what the root cause for it unless you have some minimal reproducible example.

@junqi-lu
Copy link

junqi-lu commented Jul 29, 2024

Sorry I forgot to reply to this. The most common reason that the temp json file is not valid is because the process is shut down abruptly. For example, if there's any process.kill(). There's no way for viztracer to log the files if the process is just killed. However, I don't know what the root cause for it unless you have some minimal reproducible example.


update:

hmm, it seems the issue comes from windows. I can't reproduce it on linux.


@gaogaotiantian I got the same warning. The OS is windows 11. I used multiprocessing.Pool(), too. And the viztracer command is just viztracer .\demo.py.

Minimal demo is like:

import multiprocessing
import random

def worker(input):
    return random.randint(1,10)

if __name__ == '__main__':
    with multiprocessing.Pool() as pool:
        inputs = (i for i in range(10))
        nums = list(pool.imap(worker, inputs))
    total = sum(nums)
    print(total)

@TTianshun
Copy link
Contributor

Sorry I forgot to reply to this. The most common reason that the temp json file is not valid is because the process is shut down abruptly. For example, if there's any process.kill(). There's no way for viztracer to log the files if the process is just killed. However, I don't know what the root cause for it unless you have some minimal reproducible example.

update:

hmm, it seems the issue comes from windows. I can't reproduce it on linux.

@gaogaotiantian I got the same warning. The OS is windows 11. I used multiprocessing.Pool(), too. And the viztracer command is just viztracer .\demo.py.

Minimal demo is like:

import multiprocessing
import random

def worker(input):
    return random.randint(1,10)

if __name__ == '__main__':
    with multiprocessing.Pool() as pool:
        inputs = (i for i in range(10))
        nums = list(pool.imap(worker, inputs))
    total = sum(nums)
    print(total)

Maybe you can check this. It's possible for multiprocessing to terminate the saving process on Windows. #364 (comment)

@gaogaotiantian
Copy link
Owner

@junqi-lu , yeah multiprocessing.Pool() just won't work with VizTracer on Windows. That's how CPython works and there's little we can do about that.

@gaogaotiantian gaogaotiantian closed this as not planned Won't fix, can't repro, duplicate, stale Oct 10, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

4 participants