Move the semanticdb generation logic to compile and make sure it's shared where possible

[arturaz]

This PR optimizes and reworks the compilation pipeline, with regards to SemanticDB generation.

Pre-PR situation

• there were separate compile and semanticDbDetailed tasks.
• compile performed the compilation without semantic DB plugin.
• semanticDbDetailed performed the compilation with the semantic DB plugin, but did not reuse the compile's output.

Which meant that if any of these happened:

• mill cli invoked compile and then semanticDbData.
• mill BSP invoked compile and then any other task that required the semantic db, or vice-versa.

The compilation would have been performed twice, wasting CPU cycles and worsening the developer experience.

Post PR situation

Mill now smartly chooses whether compile produces semanticdb data or not. semanticdb is produced if:

• compile was directly invoked by a task that needs semanticdb.
• there is at least one BSP client that requires semanticdb to be produced.

Implementation details

Introduction of MILL_BSP_OUTPUT_DIR

Previously you could use MILL_OUTPUT_DIR environment variable to set both regular and bsp mill's output directory to a certain folder.

Because regular mill now needs to know the location of the BSP folder, having one variable is problematic:

• you run MILL_OUTPUT_DIR=out_bsp ./mill --bsp ...
• you want to run regular mill, but provide it a changed path for bsp mill.
• MILL_OUTPUT_DIR=out_bsp ./mill ... changes the regular mill out folder.

Thus MILL_BSP_OUTPUT_DIR is introduced, which allows you to:

• MILL_BSP_OUTPUT_DIR=out_bsp ./mill --bsp ...
• MILL_BSP_OUTPUT_DIR=out_bsp ./mill ... # this still uses the regular out/ folder, but knows where bsp mill out/ folder is located

BuildCtx.bspSemanticDbSessionsFolder

Folder in the filesystem where Mill's BSP sessions that require semanticdb store an indicator file (name = process PID, contents are irrelevant) to communicate to main Mill daemon and other BSP sessions that there is at least one Mill session that will need the semanticdb.

The reasoning is that if at least one of Mill's clients requests semanticdb, then there is no point in running regular compile without semanticdb, as eventually we will have to rerun it with semanticdb, and thus we should compile with semanticdb upfront to avoid paying the price of compling twice (without semanticdb and then with it).

CompilationResult.semanticDbFiles

Because we can't change the return type of compile due to binary compatibility, semanticDbFiles field was added to CompilationResult and compile fills it in if the compilation happened with semanticdb enabled.

Removal of CompileFor and related tasks

These are not needed anymore with the single compile task.

Removal of separate SemanticDbJavaModule.semanticDbDataDetailed task

It's functionality was merged to compileInternal, which takes a compileSemanticDb parameter.

There also was a lot of code duplication between compile and semanticDbDataDetailed tasks, for both java and scala modules.

Replace the implicit Task.dest usage in ZincWorker with explicit compileTo argument

This makes it clearer what the parameter is used for and allows to reuse the same value in SemanticDbJavaModule.enhanceCompilationResultWithSemanticDb invocation.

Misc changes

• Moved JavaModule#resolveRelativeToOut instance method to UnresolvedPath.resolveRelativeToOut.
• Improved Server to provide better debugging output if the server cannot be launched.
• testScala212Version updated to 2.12.20 because new semanticdb plugin is not provided for the ancient 2.12.6 version that was used.

Things to discuss

A side-effect of this PR is that now compile always depends on semanticDbEnablePluginScalacOptions:

      if (compileSemanticDb) {
        // Filter out -Xfatal-warnings to avoid semanticdb from failing the build.
        allScalacOptions().filterNot(_ == "-Xfatal-warnings") ++
          semanticDbEnablePluginScalacOptions() ++ semanticDbOptions
      } else allScalacOptions()

Which means that Mill's task planner will run it, even if compileSemanticDb is always false.

Which then can result in errors like this:

arturaz@razor:~/work/mill> ./mill contrib.playlib.test.testForked mill.playlib.PlaySingleApiModuleTests.compile mill.playlib.PlayModuleTests.compile
[2483/2483] contrib.playlib.test.testForked
[2483] Running Test Class mill.playlib.PlaySingleApiModuleTests
[2483] --------- Running Tests mill.playlib.PlaySingleApiModuleTests.compile ---------
[2483] semanticDbEnablePluginScalacOptions task failed
[2483] X mill.playlib.PlaySingleApiModuleTests.compile 1653ms 
[2483]   scala.MatchError: Left(java.lang.RuntimeException: 
[2483]   Resolution failed for 1 modules:
[2483]   --------------------------------------------
[2483]     org.scalameta:semanticdb-scalac_2.13.12:4.13.9 
[2483]          not found: /home/arturaz/work/mill/out/contrib/twirllib/publishLocalTestRepo.dest/org/scalameta/semanticdb-scalac_2.13.12/4.13.9/semanticdb-scalac_2.13.12-4.13.9.pom
[2483]          not found: /home/arturaz/work/mill/out/libs/daemon/client/publishLocalTestRepo.dest/org/scalameta/semanticdb-scalac_2.13.12/4.13.9/semanticdb-scalac_2.13.12-4.13.9.pom
[2483]          not found: /home/arturaz/work/mill/out/libs/daemon/server/publishLocalTestRepo.dest/org/scalameta/semanticdb-scalac_2.13.12/4.13.9/semanticdb-scalac_2.13.12-4.13.9.pom
...

Which can be surprising, as "I am just running compile, why is this searching for semanticdb?".

Unfortunately, as protected def semanticDbEnablePluginScalacOptions: T[Seq[String]] is exposed in public binary API, we can't change the method signature to protected def semanticDbEnablePluginScalacOptions: Task[() => Seq[String]] so we could actually defer the running of the task.

One option is to crawl https://repo1.maven.org/maven2/org/scalameta/ and build a baked-in map of Scala version to last supported semanticdb-scalac version and use that instead of SemanticDbJavaModuleApi.buildTimeSemanticDbVersion.

Which also raises the question: why does default implementation of semanticDbVersion chooses built in semanticdb, even if user provides SEMANTICDB_VERSION environment variable?

---

Fixes: #5744