toward coulomb in scala 3

[erikerlandson]

Moving discussion on #230 here

[erikerlandson]

@armanbilge @cquiroz

Regarding source compatibility: I don't think it will be possible to move from scala-2 coulomb to scala-3 coulomb without making some modifications. From a library user perspective, I do not think coulomb will look very different, but some things are going to change:

• details of at least some import statements will change to reflect some library refactoring that I hope will simplify overall project organization
• types %*, %/, %^ will be simplified to *, /, ^: I may define aliases to aide in backward compatibility here

I'm interested in the possibility of compiling "scala-2 coulomb" for scala-3 via compiler flags. IIUC @cquiroz experimented with this and ran into some difficulties, but if we can make it work I think it would be a good feature for scala-2 coulomb, and library users who would like to change their code as little as possible while porting.

My personal philosophy regarding the move to scala 3 is that I want to take advantage of the wide variety of new language features, both for my own desire to learn them, and because simply compiling legacy scala-2 code for scala-3 (even if it works) seems to be missing the point, imo. Furthermore, moving into scala 3 seems like a good transition to introduce significant breaking changes, since the language itself is also altering substantially.

[cquiroz]

I tried to make the current coulomb compile in scala 3 but hit a brick wall to some extent with compile time ops but more with shapeless

It would be great to be able to cross compile event if you have to import different libs for scala 2 and 3

OTOH if it is going to take a lot of work it is probably not worth it, especially if it would hinder making good use of scala 3 features

[armanbilge]

I do not think coulomb will look very different

💯 Imports and ops changes don't seem that bad at all actually. I wonder if compatibility artifacts can be provided for scala 2/3 as necessary to provide aliases etc.

I'm interested in the possibility of compiling "scala-2 coulomb" for scala-3 via compiler flags.

I very briefly looked into this and it seems hard, particularly due to the dependence on non-crossed libs like shapeless-2 and singleton-ops.

Furthermore, moving into scala 3 seems like a good transition to introduce significant breaking changes, since the language itself is also altering substantially.

Intuitively, this seems like a good idea (and is certainly tempting!), but in practice can be quite challenging for downstreams. See commentary in etorreborre/specs2#1007 (comment) and typelevel/cats#3757 (comment). In fact, Scala 3 itself decided against this strategy by re-using the Scala 2.13 standard library; the "real" Scala 3 stdlib will be introduced at some future date despite the cost of another breaking change.

With that said, given the requirements/constraints I don't think this strategy is unreasonable for coulomb.

[erikerlandson]

Regarding sbt versus mill:

My interest in mill comes down to the fact that I (personally) simply do not like working with sbt. I've been using sbt for several years and I have never felt like I really understood what I was doing with it. I've discussed this topic previously, so I'll just link to that:
#162 (comment)

That said, @armanbilge if your sbt-typelevel plugin can provide what I want and is simpler to work with than my current sbt plugin hackery, I'm willing to consider it.

[armanbilge]

Done:

• Setup sbt-typelevel build #234

Quite nice if I do say so myself 😉

[erikerlandson]

@armanbilge I like the look of it, and I'm inclined to merge it

• it modernizes my CI in ways I've wanted for a while, not least mima checking
• I'm guessing it will align coulomb with the build/publish for other typelevel projects
• it looks pretty clean and reasonable to maintain

I have a few questions:

1. I see it requires some github secrets:

  PGP_PASSPHRASE: ${{ secrets.PGP_PASSPHRASE }}
  SONATYPE_PASSWORD: ${{ secrets.SONATYPE_PASSWORD }}
  SONATYPE_USERNAME: ${{ secrets.SONATYPE_USERNAME }}
  PGP_SECRET: ${{ secrets.PGP_SECRET }}

I can guess how to fill out sonatype stuff - what is the PGP stuff for?

2. What actually triggers a publish ? it mentions github.ref == 'refs/heads/scala3 as a case - but I don't think I want a new publish to kick off every time I merge or push something to develop/scala3 branch. I'd want it to happen if I create a vx.y.z tag, tho

3. I notice it is auto-generating the github actions (and also adding copyright headers?) - which I'm down with - but I'd like to understand how and when these files are added, and/or updated. Are they like pre commit hooks? Or is there an sbt command that I use to do this before I git-commit?

4. regarding mima checking - If I wanted to push something that violated mima, what is the workflow for that? I assume it expects a new major version for such an update, but my major versions are all still 0.x.x ... While I'm in rapid iteration on scala3 can I turn mima off?

5. In the current coulomb SBT, we have some subprojects that support scalaJS and some that do not: can the tlCrossRootProject.aggregate(core, units) accommodate aggregating subprojects that do not all include JSPlatform or NativePlatform if that is necessary?

[armanbilge]

Glad to hear that! 😁

There's documentation here: https://typelevel.org/sbt-typelevel/

To answer your questions.

1. Artifacts published to sonatype/maven must be signed with a PGP key (you must have also set this up for your current release strategy). So to enable CI releases, you export a secret key and set it as PGP_SECRET with a command like this:

   gpg --armor --export-secret-keys $LONG_ID | base64

   Ideally, the key is not passphrase-protected, but if it is you can set PGP_PASSPHRASE.

2. I've currently configured it in that PR to publish on every push to the scala 3 branch as well as on tags. Note that non-tag pushes will be published as commit-hash-versioned SNAPSHOTs. Of course, totally up to you, but personally I think snapshots are great: they make changes immediately available to downstreams (e.g. I can start experimenting with snapshots in gemini-hlsw/lucuma-core) and they also test that your CI release config is working.

3. Yes, that's right, the workflows are auto-generated. You can re-generate them with manually with the githubWorkflowGenerate command from sbt. Scala steward also understand them, and will automatically re-generate them as-needed. In ordinary development they won't need to be re-generated/updated, but they sometimes do when updating sbt-typelevel, updating Scala versions, adding a new sub-module to your build, tinkering with CI-release settings, etc. If the workflows are out-of-date the CI job will fail and ask you to re-generate them.

4. See: https://typelevel.org/sbt-typelevel/faq.html#how-do-i-introduce-breaking-changes-intended-for-my-next-version

   TL;DR: sbt-typelevel sets you up with early-semantic-versioning. So when you are in 0.y.z any bump in y is considering a breaking update. I've set your base version to be 0.6, since this branch represents the beginning of the 0.6 series. Until you tag and release v0.6.0 there is nothing to compare binary-compatibility against, so MiMa is effectively disabled.

5. Yup, you can aggregate any project into tlCrossRootProject including vanilla, non-cross-projects and it will handle them appropriately 👍

Hope that all makes sense, let me know if you have other questions! :)

[erikerlandson]

Thanks! i merged it 👍

publish is failing, but it will probably work once I find my pgp key and configure the secrets.

[armanbilge]

Yes, I just made an issue about it. I forgot to mention, for snapshot publishing you only need the sonatype secrets. PGP keys are only needed for real releases.

[erikerlandson]

@armanbilge, so on my local machine, if I try a basic sbt compile I get a bunch of non-fuzzy error messages:

[eje@localhost coulomb]$ sbt compile
Thread[JVMCI CompilerThread1,9,system]: Compilation of jdk.internal.loader.URLClassPath$JarLoader.getResource(String, boolean) failed: java.lang.AssertionError: dependencies invalid Code installation failed: dependencies invalid
Failed dependency of type abstract_with_unique_concrete_subtype
  context = *jdk.internal.loader.Resource
  class   = jdk.internal.loader.URLClassPath$JarLoader$2
  witness = sun.net.www.protocol.jrt.JavaRuntimeURLConnection$1

	at jdk.internal.vm.ci/jdk.vm.ci.hotspot.HotSpotCodeCacheProvider.installCode(HotSpotCodeCacheProvider.java:130)
	at [email protected]/org.graalvm.compiler.core.target.Backend.createInstalledCode(Backend.java:214)
	at [email protected]/org.graalvm.compiler.hotspot.CompilationTask.installMethod(CompilationTask.java:370)
	at [email protected]/org.graalvm.compiler.hotspot.CompilationTask$HotSpotCompilationWrapper.performCompilation(CompilationTask.java:191)
	at [email protected]/org.graalvm.compiler.hotspot.CompilationTask$HotSpotCompilationWrapper.performCompilation(CompilationTask.java:98)
	at [email protected]/org.graalvm.compiler.core.CompilationWrapper.run(CompilationWrapper.java:169)
	at [email protected]/org.graalvm.compiler.hotspot.CompilationTask.runCompilation(CompilationTask.java:330)
	at [email protected]/org.graalvm.compiler.hotspot.HotSpotGraalCompiler.compileMethod(HotSpotGraalCompiler.java:144)
	at [email protected]/org.graalvm.compiler.hotspot.HotSpotGraalCompiler.compileMethod(HotSpotGraalCompiler.java:111)
	at jdk.internal.vm.ci/jdk.vm.ci.hotspot.HotSpotJVMCIRuntime.compileMethod(HotSpotJVMCIRuntime.java:524)

[erikerlandson]

OS is fedora 35.

[eje@localhost coulomb]$ java -version
openjdk version "11.0.14" 2022-01-18
OpenJDK Runtime Environment 18.9 (build 11.0.14+9)
OpenJDK 64-Bit Server VM 18.9 (build 11.0.14+9, mixed mode, sharing)

[armanbilge]

Oh, for good measure, if you haven't done so recently try updating your sbt installation as well?

[erikerlandson]

I have other sbt projects where I see some of these weird errors. They don't seem to actually cause targets to fail, but I'd be happier if I could make them go away, or at least figure out what's causing them

[erikerlandson]

I did just update to latest https://github.com/paulp/sbt-extras
That did seem to remove a few of the errors

[armanbilge]

Oh, interesting. I don't know anything about that unfortunately :)

Nowadays I do nearly all of my development in ephemeral cloud instances. It helps with juggling various envs without mucking up my local.

[erikerlandson]

@armanbilge separate issue, when I do sbt console my project defs are not there:

[eje@localhost coulomb]$ sbt console
[info] welcome to sbt 1.6.2 (Red Hat, Inc. Java 11.0.14)
Thread[JVMCI CompilerThread2,9,system]: Compilation of jdk.internal.loader.URLClassPath$JarLoader.getResource(String, boolean) failed: java.lang.AssertionError: dependencies invalid Code installation failed: dependencies invalid
Failed dependency of type abstract_with_unique_concrete_subtype

... more scary error messages

[info] loading project definition from /home/eje/git/coulomb/project
[info] loading settings for project root from build.sbt ...
[info] resolving key references (10607 settings) ...
[info] set current project to root (in build file:/home/eje/git/coulomb/)
Welcome to Scala 3.1.1 (11.0.14, Java OpenJDK 64-Bit Server VM).
Type in expressions for evaluation. Or try :help.
                                                                                                                                                                                                                                              
scala> import coulomb.*, coulomb.conversion.standard.given, coulomb.ops.standard.given, coulomb.units.si.{*,given}, coulomb.units.si.prefixes.{*,given}
-- [E006] Not Found Error: -----------------------------------------------------
1 |import coulomb.*, coulomb.conversion.standard.given, coulomb.ops.standard.given, coulomb.units.si.{*,given}, coulomb.units.si.prefixes.{*,given}
  |       ^^^^^^^
  |       Not found: coulomb

longer explanation available when compiling with `-explain`
1 error found
                                                                                                                                                                                                                                              
scala> 

[armanbilge]

Ah, this one I know. Now I see why you don't like sbt :)

Running sbt console implicitly takes you into the console for the root project e.g. sbt root/console, which doesn't contain anything actually. It "aggregates" the other projects (such that commands like compile automatically run on all the aggregated projects) but it itself is effectively empty.

You should try something like sbt coreJVM/console or sbt unitsJVM/console. This will get you a console that includes all the stuff for that project and its dependencies.

Let me know if that makes sense 😅

[erikerlandson]

ah right, I remember this now!
Well, I also had to create an "all-subprojects" target in mill too, so doing the same here is no loss

[erikerlandson]

@armanbilge most of the content in the coulomb "microsite" is really just the output of unidoc plugin. Are there any examples of adding scaladoc to TypelevelSitePlugin ?

[erikerlandson]

I actually think dumping them to sonatype/maven and accessing via javadoc.io could be good.

• Like you say, it makes all versions permanent
• Also, I bet it makes it easy to link to multiple doc versions in the microsite, which is a thing I've sort of wanted for a while but wasn't sure how to do easily

[erikerlandson]

I am unfamiliar with mdoc, but happy to learn. Looks like mostly markdown?

[armanbilge]

Also, I bet it makes it easy to link to multiple doc versions in the microsite, which is a thing I've sort of wanted for a while but wasn't sure how to do easily

Yes, exactly this! We're trying to build-in support for versioned documentation which are quite common.

• Versioned documentation typelevel/sbt-typelevel#100

[erikerlandson]

that sounds great! I don't need to get into scaladoc publishing immediately, I can just write it, and probaby by the time it matters you'll have some of this working

[armanbilge]

I am unfamiliar with mdoc, but happy to learn. Looks like mostly markdown?

The magic of mdoc is not the markdown actually (I think it ignores it) but that it actually runs scala code you put in your code blocks. So you can have documentation that is checked in CI and shows meaningful output to users. Here's an example:
https://typelevel.org/mouse/#example

All those commented outputs are auto-generated!

[armanbilge]

Scala 3 inline and macros community census

https://contributors.scala-lang.org/t/scala-3-inline-and-macros-community-census/5608

@erikerlandson since you are making heavy use of these in coulomb I think your feedback could be very valuable :)

[erikerlandson]

@armanbilge @cquiroz
I wrote some unit tests in JavaTimeSuite that break the JS linker, I guess it can't resolve stuff in java.time.xxx
I can't remember if there is some clever way to have test suites that only build/run in rootJVM, or if I just have to pull it out into a different subproject

[erikerlandson]

I forget the directory structure for putting jvm specifc (or js specific) code. is it shared, jvm, js ?

[armanbilge]

Yes, that's right. Here's an example:
https://github.com/typelevel/feral/tree/main/lambda

You also need to remove CrossType.Pure from any projects that use this split directory structure.

coulomb/build.sbt

Line 34 in 9d0b619

.crossType(CrossType.Pure)

[erikerlandson]

tried adding:

(units.js / libraryDependencies) += "io.github.cquiroz" %%% "scala-java-time" % "2.4.0-M2" % Test

Which resolves and compiles, but the JS tests still throw link errors

Referring to non-existent class java.time.Duration
Referring to non-existent class java.time.Instant

[armanbilge]

@erikerlandson do you have a branch/PR I can look at?

[erikerlandson]

#263