Fix CPU starvation warning in MillBuildTool classpath extraction#17
Fix CPU starvation warning in MillBuildTool classpath extraction#17
Conversation
parseClassesDir and ClasspathOutputParser.parseJsonArray perform Files.isDirectory/Files.exists calls that were running on the compute pool, triggering cats-effect CPU starvation warnings. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
In AllSymbolsStream and SymbolLister, the .filter(PublicApiFilter.isPublic) calls can trigger lazy TASTy deserialization in tasty-query when accessing isPrivate/isSynthetic flags. Use evalFilter with IO.blocking instead. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
There was a problem hiding this comment.
Pull request overview
Adjusts execution context usage to avoid cats-effect CPU starvation warnings by moving file-system–backed classpath extraction work off the compute pool.
Changes:
- Wrap
parseClassesDir(includesFiles.isDirectory) inIO.blockingduring Mill classpath extraction. - Wrap
ClasspathOutputParser.parseJsonArray(includesFiles.existschecks) inIO.blocking. - Change symbol-stream filtering to use
evalFilterwithIO.blocking(PublicApiFilter.isPublic).
Reviewed changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated 3 comments.
| File | Description |
|---|---|
lib/src/cellar/build/MillBuildTool.scala |
Moves blocking filesystem checks involved in classpath extraction onto the blocking pool. |
lib/src/cellar/SymbolLister.scala |
Switches public filtering to effectful evalFilter with IO.blocking. |
lib/src/cellar/AllSymbolsStream.scala |
Switches public filtering to effectful evalFilter with IO.blocking. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| .eval(IO.blocking(pkg.declarations)) | ||
| .flatMap(decls => Stream.emits(decls)) | ||
| .filter(PublicApiFilter.isPublic) | ||
| .evalFilter(sym => IO.blocking(PublicApiFilter.isPublic(sym))) |
There was a problem hiding this comment.
PublicApiFilter.isPublic is a pure, in-memory predicate (see lib/src/cellar/PublicApiFilter.scala) and does not perform blocking I/O. Wrapping it in IO.blocking and switching from .filter to .evalFilter introduces an effect + blocking-pool shift per symbol, which can significantly hurt throughput when listing many symbols. Prefer keeping .filter(PublicApiFilter.isPublic) (or, if an effect is truly needed, use evalFilter(sym => IO(PublicApiFilter.isPublic(sym))) without blocking).
| .eval(IO.blocking(SymbolResolver.collectClassMembers(cls))) | ||
| .flatMap(syms => Stream.emits(syms)) | ||
| .filter(PublicApiFilter.isPublic) | ||
| .evalFilter(sym => IO.blocking(PublicApiFilter.isPublic(sym))) |
There was a problem hiding this comment.
Same concern here: PublicApiFilter.isPublic is not blocking, so IO.blocking + evalFilter adds unnecessary overhead and shifts every element to the blocking pool. Consider reverting to .filter(PublicApiFilter.isPublic) (or evalFilter(sym => IO(...)) without blocking if you need an effect for composition).
| .flatMap(syms => Stream.emits(syms)) | ||
| } | ||
| .filter(PublicApiFilter.isPublic) | ||
| .evalFilter(sym => IO.blocking(PublicApiFilter.isPublic(sym))) |
There was a problem hiding this comment.
This change moves public-symbol filtering onto IO.blocking, but PublicApiFilter.isPublic is a pure predicate (no file-system calls). Besides the per-element context shift overhead, this is also a discrepancy with the PR title/description which focus on Mill classpath extraction; either document why symbol filtering needed this change (and how it addresses CPU-starvation warnings), or split/revert this part to keep the PR scoped.
Summary
parseClassesDir(callsFiles.isDirectory) inIO.blocking— was running as a pure assignment on the compute poolClasspathOutputParser.parseJsonArraycall (iteratesFiles.existsover all classpath entries) inIO.blocking— was pattern-matched directly on the compute poolThese blocking file-system checks on the compute thread pool triggered the cats-effect CPU starvation warning:
Test plan
./mill lib.compilepassesget --module lib cellar.build.MillBuildToolreturns results without warningsearch --module lib ProcessRunnerandlist --module lib cellar.buildwork cleanlyget-external org.typelevel:cats-effect_3:3.5.4 cats.effect.IOworks🤖 Generated with Claude Code