Java files missing from db after extraction using gradle on M1 Mac

[s0]

Hi There,

Former Semmle / GitHub employee here 👋

I'm currently doing some demonstration / evaluation work on a Java 11 codebase that is build using gradle. I'm using the CodeQL CLI on a new Macbook Pro with an M1 Pro, and creating the database with codeql database create seems to work fine, and the build completes... however there are a number of Java files missing (around a third to a half) from the database that's produced.

A bit more information:

• The relevant .class files are produced in their build directories, (which means they're getting compiled).
• I've tried both with the autobuilder, and with a specific command ensuring --no-daemon is passed, same result.
• I've observed the same behaviour on a few different codebases of varying size.

I imagine that there's something strange going on here with CodeQL's instrumentation of the build, but I'm scratching my head as to what.

[hmakholm]

Hi Sam!

Build tracing on M1 Macs has been a bit of a challenge, mostly due to the instrumentation needing to work with a mixed-architecture process tree. We thought we had gotten it to work with the most recent releases of CodeQL, but it is possible there are still corner cases where something goes wrong. On the other hand, it sounds a bit strange that it would be only some of the files that go missing, rather than all of them. Are the missing files scattered randomly throughout the source tree, or concentrated in particular subdirectories?

The codeql database create command should have left some log files in the log subdirectory which can be help for troubleshooting the missing files from the extraction.

In particular, you can try grepping through build-tracer.log and javac-extractor-*.log for references to the files that are missing from the completed database. If they're not mentioned in both log files, that's interesting information too. If they are mentioned, then here might be useful diagnostic messages nearby, though they are not guaranteed to make perfect sense for outsiders.

If you don't find any obvious problem you can handle yourself, can you open an "issue" on this repo rather than this "discussion" (issues are easier to assign to engineering teams for looking into), and attach a zip of the log directory from a database create that exhibits the problem, please? Please also note the names and paths of one or more of the files that are missing.

[s0]

Thanks @hmakholm for the pointers. Looking at the logs was very helpful.

It looks like it was actually unrelated to M1 macs, and the missing files were due to use of Lombok. Following the advice on #4984 allowed me to produce a complete database. Thanks!