Request for Assistance in Collecting Code Coverage for JIT in PyPy

[zzl261004]

I am interested in collecting coverage data for the rpython/jit source code while executing programs with PyPy. Specifically, I would like to know which parts of the code in rpython/jit are being used during program execution. Could you please advise on what tools or methods are recommended for this purpose?

[mattip]

Do you mean the internal PyPy source code or the code of your program being run with PyPY? For the former, there is no good answer. For the latter, see vmprof-python and the jit log documentation

[zzl261004]

I mean the internal PyPy source code.For example, gcov can be used to collect the coverage of cpython.

[mattip]

Using gcov only gives you part of the story. There is a lot of pure-python code (in lib-python and lib_pypy) that is used when running the interpreter, this code is interpreted and sometimes JITted during execution. How would you get coverage statistics on that? How does your analysis of CPython handle the stdlib code?

You could theoretically use gcov with PyPy as well, but then you would need a reverse map of the RPython-to-C-code. Perhaps resolving #4837 would help in that direction, help is needed.

What would you do with that data once you have it?

[cfbolz]

Maybe it would help us a little bit to understand what exactly this is for? From the bugs you report it sounds like you're working on fuzzing the jit, maybe? Do you want to try to use some kind of coverage guided fuzzing?

I'm sure it would be possible to use gcov with pypy by hacking the automatically generated Makefile (in /tmp/usession-*/testing_1/Makefile). However, that gives you coverage of the interpreter and the meta-jit only, not really any information about which part of the interpreter the meta-jit is compiling.

Getting coverage information about what part of the interpreter the jit is compiling could be interesting! That would require a bit of work on the rpython internals though. If you want to go in this direction, I'm happy to provide pointers.

[zzl261004]

Yes, I want to try using coverage-guided fuzzing and focus on the JIT source code. Perhaps I could try instrumenting the RPython code of the JIT to obtain coverage information, such as function coverage. For example, which parts of the functions are being compiled by the JIT? Could you please provide me with some guidance on this?

[cfbolz]

ok. which fuzzer are you using? one of the afl-based ones?

[zzl261004]

Thanks. I did not use any existing fuzz testing methods. I am writing a coverage-guided fuzz testing method.