Got to the bottom of a stubborn problem trying to export Coulomb

[benhutchison]

exporting Coulomb has been very flaky for me, lots of weird compiler errors in application code that tries to use Coulomb in exported form. I finally got to the bottom of why.

The apparent workaround is to avoid export coulomb.Quantity and instead alias via type Quantity[U, V] = coulomb.Quantity[U, V]

Copied from the Scala discord:

severian — Today at 6:41 PM
I've hit an issue using Scala 3 export. There seems to be an ambiguity (or double-meaning) in the current syntax rules [https://docs.scala-lang.org/scala3/reference/other-new-features/export.html].

Say I have a package that defines a type and an object:

package exporter

opaque type Foo = Int
object Foo:
  def bar = ???

If I then export this via the expression export exporter.Foo, it seems that I export both

• the type Foo
• the object Foo

Sometimes this is undesirable. I cannot find any export syntax that can refer to solely the type, or the object members, without exporting the other one. There's an unwanted "pun" in the export syntax. Does anyone know a way?

BTW this is not a purely academic question. I have spent several days tracking down a subtle issue when exporting the Coulomb [https://github.com/erikerlandson/coulomb] library that is caused by the double-export behavior.

[erikerlandson]

when you say it exports bar, is it still scoped to the companion object? that is, do you still have to call Foo.bar ?

when I was working with the opaque type design for Quantity, I also found that its behavior was a bit unintuitive. I also recall having problems exporting objects that are defined at a package level. I define anything I want to export inside of an object, which seems to work better.

[benhutchison]

You are indeed correct, in my example, it exports the type Foo and the object Foo. I've updated my comment above to avoid confusing future readers.

Actually, "got to the bottom of a stubborn problem" was an overstatement I see. What I have found is a way to export Coulomb Quantity that seems to work, (type Quantity[U, V] = coulomb.Quantity[U, V] and don't name it in the package export clause).

But I don't understand why the export of the Quantity object causes a compile error. There's another condition needed to trip the error, that I discovered through experiment: it happens when the application level quantity, eg Quantity[BigDecimal, MyAssetUnit] is defined as a type alias.

In such cases, when invoked the inlined math operations like +, the Quantity type gets somehow "double wrapped".

You can see an example in the Scastie. Its looking for AdditiveSemigroup[coulomb.Quantity[Int, use.MyUnit]] but that's just wrong; it ought to need a AdditiveSemigroup[Int] right?

[erikerlandson]

yes, it does seem to be binding the entire Quantity value into V, instead of just Int. I took a look at your code, and at the moment I don't see why that would be happening.

[erikerlandson]

Here is my theory about what's going on. Quantity[V, U] is an opaque type for V - so I think that maybe the type system is, somewhere in this process, mapping back from V to Quantity[V, U]. So it matches V properly but then the AdditiveSemigroup is using Quantity[V, U] because to scala it's the same as V.

@armanbilge does that seem plausible?

[erikerlandson]

One thing that might work to work around some of this is if I change ALL of the definitions in coulomb to be inside object instead of package - there are no guarantees but overall I have had more success with export from object, and it's conceivable that this would make your export scheme work better.

It would break MiMa, and might require some of the path names to change. The advantage of package is that it can be shared across source files. object can't and that would change things.

[erikerlandson]

Another change I might make would be to simply give up on using opaque types. Overall, they aren't as helpful as I had hoped, in terms of baking down to raw types in the byte-code. In reality they are boxed in many contexts, and so simply going back to class Quantity[V, U] might be worth it, if it reduces some of these scala type system edge cases

[erikerlandson]

xref about opaque boxing: scala/scala3#12009

[erikerlandson]

re-defining Quantity as non-opaque would be a much easier change than trying to replace package with object

[erikerlandson]

either would break mima checking

[erikerlandson]

I tried a simplified reproducer of what I believe to be the basic structure of coulomb, and for what it's worth, it seems to work. There are a lot of advanced scala features in play with coulomb, and it's hard to know if any candidate reproducer is capturing the right ones.

package freetp {
    opaque type FreeTP[V, U] = V
    object FreeTP:
        def apply[U](using a: Applier[U]) = a

        class Applier[U]:
            def apply[V](v: V): FreeTP[V, U] = v
        object Applier:
            given given_Applier[U]: Applier[U] = new Applier[U]

        extension [V, U](f: FreeTP[V, U])
            def value: V = f
}

package exporter {
    object all:
        // no wildcard exports from a package
        export _root_.coulomb.freetp.FreeTP
        // ok to wildcard export from object
        export math.{*, given}

    object math:
        import _root_.coulomb.freetp.FreeTP

        abstract class AddFreeTP[V, U]:
            def plus(x: FreeTP[V, U], y: FreeTP[V, U]): FreeTP[V, U]

        given given_AddFreeTP[V, U](using n: Numeric[V]): AddFreeTP[V, U] =
            new AddFreeTP[V, U]:
                def plus(x: FreeTP[V, U], y: FreeTP[V, U]): FreeTP[V, U] =
                    FreeTP[U](n.plus(x.value, y.value))
}

object repro:
    final type Free
    import _root_.coulomb.exporter.all.{*, given}
    val x = FreeTP[Free](1)
    val add = summon[AddFreeTP[Int, Free]]
    val x2 = add.plus(x, x)

[erikerlandson]

It may also have to do with defining that extension inside of the FreeTP companion object, where it bridges the world of the type name and its opaque alias. I recall doing that because I was having issues with the extension methods being importable.

Possibly this is another argument for avoiding opaque type.

[erikerlandson]

Hah! I think this is a compiler bug. If I try to reference .value it errors out!

object repro:
    final type Free
    import _root_.coulomb.exporter.all.{*, given}
    val x = FreeTP[Free](1)
    val x2 = x + x // scala types 'x2' as Int !!
    val v = x2.value  // hah, this is a bug! it fails to compile!

[erikerlandson]

I've been playing around with trying to minimize it further, and this reproducer really does seem to be minimal. It happens with opaque type + transparent inline + export

If you import directly (without export) it types correctly:

object repro:
    final type Free
    import _root_.coulomb.exporter.all.{*, given}
    val x = FreeTP[Int, Free](1)
    val x2 = x + x // scala types 'x2' as Int !!

object repro2:
    final type Free
    // don't use exporter.all
    import _root_.coulomb.freetp.FreeTP
    import _root_.coulomb.exporter.math.{*, given}
    val x = FreeTP[Int, Free](1)
    val x2 = x + x // this is typed correctly

[erikerlandson]

Filed a compiler bug issue:
scala/scala3#17151

[erikerlandson]

Filed another compiler bug:
scala/scala3#17220

This one may be the same underlying bug as scala/scala3#17151, but it manifests a bit differently.

[benhutchison]

Hi @erikerlandson

1. I'm again in awe of your programming skills, this is a masterful minimization of a very subtle issue. I always felt there's a compiler bug rattling about here.

2. I read your xrefs on opaque types. That was interesting, if somewhat heartbreaking, reading.

   I recall reading some industry excitement (not EPFL to be clear) that had dubbed Opaque Types as "zero cost abstraction, finally" but alas that proves a false hope. Instead, opaque types go to the Crypt of Dissappointing Optimizations, to be laid alongside Scala 2 Specialization and Scala 2 Value Classes 🤦

   Despite that, my vote would be to retain opaque types in Coulomb if at all feasible. I feel like they are still the theoretically correct approach.

   Scala 3 is young but has a long future ahead of it. Some form of compiler-generated specialization could yet make a comeback. Or even, the JVM JIT just sees through the boxing bytecode and realizes it can discard it. And Valhalla, now imminent, will surely only increase the impetus for more efficient handling of primitives.

   I honestly think anyone building on Coulomb has to be thinking longer-term for their ROI anyway. Lets face it, the learning curve of introducing ubiquitous Units-of-measure is considerable.

[erikerlandson]

I do agree that opaque type is the philosophically right answer - the unit type parameter U is "pure context" - it is for the type system's use, and only the raw values of type V should ever appear in byte code. In a perfect world, an expression like 1.0.withUnit[Kilo*Meter].toUnit[Meter] could always compile to a raw multiplication in byte code:

1.0 * 1000.0

In the future, with project Valhalla, and a scala-3 that can make full use of it, maybe we can get closer to this ideal state of affairs.

It concerns me that there are edge cases like the above in scala's type system, but hopefully they can fix whatever that is.

[erikerlandson]

the learning curve of introducing ubiquitous Units-of-measure is considerable

I admit that I feel as though I failed here - I really wanted coulomb to be easy to use 😂

[benhutchison]

I think much of the learning curve is structural, so I wouldn't take it to reflect on Coulomb.

It's just a huge mental/cultural adjustment to put all numbers into separate "swim lanes" by what they quantify. My experience so far is that it tends to reveal a bunch of bugs / dubious code that needs to be thought about. One example is just treating variance as having the same dimension as the underlying values it's derived from.

Also, migrating onto UoM on existing codebase has a huge surface area, it's almost a complete rewrite. I'm trialing this ATM and Im finding I have to go module by module, test by test. The consolation is that I'm learning a lot along the way.