Publish defs (and generators?) for popular web component libraries, e.g. UI5, Material

[armanbilge]

I think the new source-generation approach has been a big success. I am really really happy with how we've been able to use it in Calico. Thank you!

Building on that, I think an interesting next step would be to start aggregating and publish defs for web components. At some level, web components are very similar (or even basically the same) as HTML. So a lot of the machinery is already in place here, and it would mean we can pool our efforts to support web components as as well.

The laminar-web-components project is already doing basically this:
https://github.com/uosis/laminar-web-components/blob/cf9fdc33ae0e77a2eb5b169acc4bf8e4d89dd48c/generator/material.sc

[raquo]

So...

1. We need a data format to represent web component features – attrs, methods, method params, etc.
2. We have N number of Scala.js libraries, that we would need to generate web component interfaces for
3. We have M number of JS web component libraries out there, that we need data for (all the attrs, their types, etc.)

Then, if SDT hosted (1), and individual Scala.js UI libraries hosted (2), we could turn this N*M sized problem into an N+M sized problem. (For reference, for the regular HTML types (not web components), SDT only hosts (1) and (3), and we delegate (2) to the individual Scala.js UI libraries.)

What we get in return is potentially very good coverage for web component support in various Scala.js UI libraries, at a reasonable marginal cost. Whereas the situation right now is that interfaces to Web Component libraries are created for specific Scala.js UI libraries because usually the author only cares about one UI library that they use.

In practice this could look like this:
A) SDT hosts a web component traits generator as well as the types needed to encode all the related source data
B) Scala.js UI libraries (or at least their authors, perhaps in a separate project) host configurations of SDT generators that produce web component code for this UI library using SDT. Note that these generators don't yet have access to web components data to generate the definitions from.
C) People who want to create web component definitions for Scala.js would somehow compile the web components type data in the format prescribed by (A), then they will pass that data to the generator(s) in (B) above to generate the UI library-specific interfaces for web components and publish those artifacts.

Although technically this is rather convoluted, I think it will scale better socially as the division of responsibilities better matches the interests of all the people involved. This will require a lot of work, especially the compiling-web-components-type-data part, so this will probably scale better than just having SDT act as a centralized repository for multiple web component libraries data.

---

I do worry about (1) above. Web components can have arbitrary methods on them returning arbitrary params and returning arbitrary types. That all can be very dynamic because it's JS under the hood. The generator would need to be expressive enough to support all that.

While some web component libraries publish a manifest of all their attributes etc., it (sometimes? usually? always?) does not have enough type information to generate scala types straight out of it, but it could be used as a starting point I guess.

[armanbilge]

Web components can have arbitrary methods on them returning arbitrary params and returning arbitrary types.

Hmm, I'm confused about this bit. HTMLElement and friends also have various "arbitrary" methods on them, but SDT doesn't take responsibility for that. The expectation is that the user will use scala-js-dom facades to access these, no?

So I was imagining the same thing here: SDT should only take responsibility for the "declarative" stuff, that is set as attributes/props/whatever. For complex interactions with the web components, users will need a facade—exactly like they do for complex interactions with vanilla HTML elements.

[raquo]

The expectation is that the user will use scala-js-dom facades to access these, no?

Those can't be in scala-js-dom because the web component's methods are specific to said web component, for example some Input web component from the material design library could have a method like .clear() exposed on it that is not available on the regular HTML element (I assume, otherwise that would be one bad example haha). Some SAP UI5 components have a few of those custom methods I think.

Basically, for this, think of any given web component type as a custom HTML element that subclasses HtmlElement.

[armanbilge]

Those can't be in scala-js-dom because the web component's methods are specific to said web component

They shouldn't go in scala-js-dom, but these components should be facaded in a library like scala-js-dom. It would be up to the user to select and provide such a library (same as it currently is for scala-js-dom, which is how I substitute it for fs2-dom).

[raquo]

I don't have too much experience with Web Components. I assume that their custom methods are actually regularly used, but I could be wrong.

I guess you can always encode the list of custom web component methods as a List[String] in the generator, without fancy types – just provide source code of the methods. This should work if you don't need to customize this part of the output, and since those methods form a js.native interface, I guess that's fine, you can't e.g. put IO in there even if you wanted to, since the interface is defined by JS.

[armanbilge]

I feel like we are talking past each other :) let's forget about web components for a minute.

If we think about vanilla HTML tags, as we know these are actually JavaScript classes like HTMLElement and stuff. Furthermore, we know that it is possible to set attributes and props on these elements, and it is also possible (and sometimes quite useful) to call methods on these elements. Right? :)

SDT helps us like this:

1. by matching tag names to their corresponding JS classes
2. by matching attributes/props to their types and codecs

Note: SDT does not do anything special to help us use the methods! The methods are not included in the SDT data/defs, and there is no facility for generating sources that would somehow call these methods.

SDT however does enable this indirectly: by matching the tag to the JS class name, the UI library can provide the user typesafe access to the underlying element. In this way the user can call methods on that element.

Furthermore, SDT is very un-prescriptive here: the UI library is free to use the facades from scala-js-dom, or their own library (as I do), or whatever.

---

Turning our attention back to web components, I propose that we treat them the exact same way. Specifically, our goal should be to:

1. match tag names to their corresponding classes
2. match attributes/props to their types and codecs

I strongly believe we should not get anywhere into the business of methods that may be defined on these web components. That is not SDT's role, and as you point out it would be difficult to do something flexible enough to capture all the necessary complexity. This is the role of some kind of facade library (like sjs-dom, or something generated by ScalablyTyped). SDT is not a facade.

---

I hope this makes sense :)

[raquo]

Yes, we're on the same page that web component method listings don't belong in SDT, although perhaps we have different reasons for agreeing on that (for me it's because I think that the web component attribute listings don't belong in SDT either, but should be in separate projects managed by other people (see (C) in my proposal)).

I understand the roles of SDT and scala-js-dom, but for web components we don't really have that distinction, because we currently don't have an equivalent to scala-js-dom for material-ui (or any other set of web components), and I don't think such an equivalent would be very useful on its own, so I don't think it's worth it to separate those in a separate facade-only repo, but that can be done if desired, that would be up to whoever implements scalajs-material-ui-bindings (and scalajs-material-ui-facade I guess if they wanted to).

I'm looking at this from the perspective of "who will maintain this?". Suppose some person wants to use material UI in some Scala.js UI library, e.g. Laminar, for one completely unbiased example. For this, they will create a scalajs-material-ui repo where they need to define all the attributes for material UI web components, and they will also need to define their methods (in one way or another, doesn't matter how it's done techncially), and they will need to pass all that data to the SDT generator (configured by the author of Laminar to emit valid Laminar code, see (B) in my proposal above) to generate Laminar code which they can publish as an artifact to end users. And since they can trivially do the same for Calico (assuming that the author of Calico also publishes an SDT generator config (B) that emits valid Calico code), perhaps that person will do that too.

So both the attribute and method definitions of the web components will end up in the same repo since it's the same person doing them. How they're encoded though I personally prefer the resulting web component definition to be contained in one single file (like this) so for that I would probably want the web component generator receive the list of methods in something like methodLines (like we currently have with headerLines / commentLines). If you want to keep method definitions separate from the rest of the web component definition, e.g. refer to some trait in another file instead of inlining the methods in trait RawElement, I guess the generator could support that as well.

---

I think... despite admittedly talking over each other (I'm just not super focused right now...) we have enough of the details in consensus that whenever someone actually has the time and will to implement this stuff, that the preferred path is clear enough to get started. I think the rest will become clear at implementation time, and that's fine, really, we don't need to do all the thinking in abstract and in advance. Cheers

[raquo]

Eh I'm re-reading my messages and it's a real mess, sorry. I think I need to actually code to express my idea but I can't do that right now. Hopefully the broad strokes are clear at least.

[armanbilge]

Not at all, thanks for engaging in the discussion :) and agree this is probably at a stage where it can be better discussed via code.

[raquo]

Spent a few days getting a head start on this whole thing. Kinda tapped out, need a break, but I'll come back to it.

https://github.com/raquo/laminar-shoelace-components/ (WIP)

This parses custom-elements.json file that is typically published by web component libraries, translates it into a scala structure that is easy to output the code from, and outputs the code. The translator is where the bulk of complexity lives. See more details in that project's README.