Create source generator to pimp our enums

[dodexahedron]

Prologue

So... As I've griped about before, enums in c# and dotnet in general just suck.

Unfortunately, they're what we have, though, in the language/SDK, and creating something with the same ease of definition, ease of use, and the same behavior is not a simple task and has plenty of potential pitfalls - not to mention it takes a ton of time and effort to do.

[Roslyn enters stage right]

Roslyn: Hi, folks! I'm here to solve all your problems and replace them with different ones!

[End scene]

So yeah...

EpiPrologue

We have plenty of enums in the project code base.

And don't get me wrong - That's not bad or wrong on the part of anyone who has made or used one, since there aren't many alternatives and the ones that do exist aren't particularly well-known (BitVector32, for example), and aren't drop-in replacements, due to other limitations or just realities of the language and SDK.

For enums as a whole, I'm not going to re-hash all the bullets I've mentioned elsewhere about what is unfortunate and insidious about them, except to say that a better way certainly is possible, especially in the areas of run-time performance and design-time quality of life.

The first generator I put in, in #3438, is a step in that direction, specifically for performance, and there are additional things I will likely still add to it, as well, unless it turns out that this obsoletes that, which it very well may do, ultimately, depending on how much I can make this one do without too much extra work.

But, Flags enums.....

Non-critical section - Expand to see complaints about enums

Flags enums, in particular, are something I very much want to make better in a lot of ways, due to runtime costs (at least without really cumbersome means of avoiding them), as well as other unfortunate stuff like:

Extension methods are common, and often for similar or the same purposes, but are also often inconsistent, and are not easily discoverable because they live in separate, sometimes multi-purpose, classes.

Conditional constructs involving Flags enums are super cumbersome, and are also something that often lead to extension methods being written to make code easier to deal with, but end up making their already bad performance even worse in many cases, usually by at least 2X (we're still talking about fast stuff though - but it all adds up).

Explicit values involving the MSB being set to 1 are really unintuitive unless the enum is unsigned...which then leads to them being defined as unsigned. And that's not bad, per se, but it does lead to inconsistency, which can be annoying in some cases and require work-arounds or result in deceptively expensive run-time behavior.

Certain rules and guidelines around design and use of enums - especially Flags enums - are not always appropriately followed, such as the definition of the default (0) value and what it means in use.

They're not actually constants in most situations.

Flag combinations are easy to mess up and hard to read, and often not in any particular order.

While extension methods make them better to use, it's still quite a bit of effort to do nice things, like fluent API implementations (such as all that work @tig did with Key-related stuff to make working with that so much better than it used to be), and that and other things often result in multiple types that each do some of what we want but aren't totally fungible, without a heap of work that is unreasonable to ask of people.

Enums, since they can't have methods defined on them, thus are not possible to cast between types that we do not have direct control over, nor (at least directly) to other enum types, which also ends up leading to methods to deal with those conversions, be they private instance methods in a class that needs them or static/extension methods somewhere.

Those issues certainly can all be solved, if someone puts in an inordinate amount of effort to design an entire type and accompanying tests and documentation that not only does what is needed, but is robust, efficient, consistent with other types, and still actually as easy to use and maintain as an enum.

So, I want to basically do that, but do it once in a source generator.

UX design goals/intentions

The basic design goals I have, for the developer experience, are simple:

• Should not be difficult to use or require advanced knowledge of specific or obscure concepts, language quirks, etc.
• Should be something that makes one prefer to use it over an enum - not just because someone asked them to.
• Should make sense
• Should have fluent API style, if and where possible and appropriate.
• Should not place additional testing burden on the developer that wouldn't exist for the same uses.
  • This requires that the generator itself have tests that prove it works correctly, similar to and beyond the ones I already wrote for the existing generator.

TG Build/Design-Time Technical requirements / expectations / limits (rough spec)

This is a list of the environment I'm writing this with the expectation of a user or system who is building/developing Terminal.Gui itself to have. These are likely to significantly loosen by the time I'm done, but shouldn't matter anyway, especially for a TG dev, and I know for a fact that the three of us meet these requirements.

• MSBuild >=17.9
  • This actually covers almost all of the requirements below, already, but I'm listing some just for formality...
• .net SDK version 8.0 or higher
• C# language version 12 or higher
• 64-bit platform
  • Which is already a Visual Studio 2022 requirement anyway
  • For those wanting to build at the command line, 32-bit .net 8 SDK is already not supported on all platforms, and 32-bit support will almost certainly continue to reduce in the future.
    • No mac 32-bit .net 8 exists, and no 32-bit MacOS exists in any current supported version.
    • .net8.0 on 32-bit Linux isn't possible on most of the major distributions' current LTS releases, and some don't even have 32-bit flavors of their distributions in the first place.
    • In short: Tough beans, and if anyone comes along who needs it that badly, they can do it themselves or just download a binary or build it on another machine.
• (Optional) If you want to use the powershell modules, PowerShell 7.4 or higher, plus anything the module manifests say are needed.
• Little-endian CPU architecture.¹

High-level use/behavior design goals (rough spec)

How do I envision the generator being told what to do, where to do it, and how?

By making it use actual enums as its input.

The design I have in mind, basically has the following workflow/behavior (remember - I want it to be basically seamless and 0-effort on the developer's part, without you having to remember to use it):

• You write an enum, just like you always have and are used to.
• An analyzer accompanying the generator looks for enums with the [Flags] attribute on it.
  • If the analyzer sees you violating certain specific rules, it will suggest you fix them (mostly the 0 thing, but probably a few others). Severity of the inspection is also, for free, configurable by the developer in the .editorconfig, DotSettings, preprocessor directives, suppression attributes, or suppression comments, like any other analyzer, so you can shut it up or even make it consider violations errors if you want.
• On the generator side of things, it will watch for the [Flags] attribute, as well.
• Enums it finds with that attribute would automatically be included for generation of a pimped out alternative type.
• To address cases where you either do not want or do not need the generator to run, suppression should be possible, either with an attribute or a comment.
  • Probably one or mayyyybe a few attributes, rather than a comment, so it can be easier to locate with Roslyn but mainly so it can take advantage of intellisense and be even more flexible than just a kill switch, with the ability to still be a full suppression or to turn off or modify specific behaviors, if any such requirements or desires come up along the way.
• If not suppressed, code will be generated that makes a struct type that is intended to be used instead of the enum.
• XmlDoc comments from the enum and enum members will be carried through to the corresponding elements of the generated code.
• Another analyzer would also be running, looking for uses of enums for which the generator made a replacement:
  • Such usages would be flagged with an appropriate diagnostic, probably at least at suggestion level, but maybe even warning, depending on how much of a difference it would make. We'll see. Again, though, that's always configurable or suppressible in the IDE itself.
  • Ideally, if it's not too much extra work to offer in a way that wouldn't cause problems, there would also be an associated codefix for at least some of the code that this analyzer flags, so you can make the switch by clicking the light bulb like you get from codefixes in VS or ReSharper etc. That is more of a stretch goal, though, especially since fixing it should be trivial anyway, because of my UX goals above.
• Probably also an analyzer to yell at you to document properly, so those warnings aren't suppressed by suppression of built-in analyzers and would have to be turned off intentionally. But this is also more of a nice-to-have and not something guaranteed to even get implemented.

So, in short, you literally would have to do zero extra work to cause these new types to be created for Flags enums, and those types would be immediately available for use even for existing code the first time you load the solution with the new analyzers/generator implemented.

And, since it's all source gen, it'll be trim-friendly and not result in additional dependencies for Terminal.Gui consumers.

Structure/Capabilities/Design of generated code (rough spec)

This part is big, because it's a combination of just a brain dump of my current ideas for implementation as well as intended to be at least a partial spec for the actual generator.

Lame humor, notes, and general intro to the big list ahead

Go take a bathroom break (or I guess read this on your phone? You do you, yo), get a drink, take a nap, or otherwise get comfy however you get comfy, before proceeding. 😝

So...
What do I plan for the generated code to do/be/have?

Here's at least a partial list.
And remember, this is for Flags enums, at the moment, which is very relevant to several of the items in the list, even if not stated in-line with them...

The top-level bullets are mostly broader concepts of how I envision it working, with sub-lists being excessive detail, explanations of my thought process/reasoning behind them, gripes about enums, justifications, or additional specs, as unnecessary.

• Types generated from your Flags enums will be public readonly partial record struct types.
  • readonly struct is the underlyng type of an enum, as that's what int, uint, etc are, even though System.Enum is a class, because the language and the compiler just define it that way and break some rules to do so that only they can break.
  • record brings many useful bits of functionality along for free.
  • partial allows the user or other source generators to extend the type even more.
  • public by default. I may include optional ability to have other accessibility modifiers. My base generator types have properties for it, but the EnumExtensionMethodsGenerator just has it return a constant, because that VERY quickly gets complex due to language rules of what has to have how high or low accessibility relative to other elements of certain types, and I didn't consider it a required feature right now. If I do allow it for this one, I'll probably leave it up to you to do it correctly, since it's not hard and the compiler will yell at you anyway until it's legal (and tell you how to fix it, even).
• Unless explicitly stated, generated structs will only allow and only produce values that can actually be represented by the defined flags.
  • For example, if you defined 5 unique flags, regardless of their actual bit positions, any attempts to assign a value to a variable of that type that includes a 1 in an undefined position is invalid.
  • Enums can have undefined values assigned to variables of their type, which almost nobody ever accounts for in code dealing with enums and can result in weird/undefined behavior or crashes.
  • Possibilities for how the generated code reacts to that could be either an appropriate exception being thrown or the value simply being masked by the defined flags and only actually setting defined bits to 1. Haven't decided yet. This may be a good option for a configurable option, if there's desire for it. Either way, it's more predictable behavior at run-time than an enum has. 🤷‍♂️²
• Generated structs will be treated as bit vectors, because that's what a Flags enum is.
  • In other words, signed or unsigned does not matter to the generated type, and the value is literally just the 32 binary bits that it is stored in.
  • This also means there's no point in explicitly declaring int or uint backing types for your flags enums.
• Numeric types or enum types used as operands with them will also be treated as bit vectors.
• They will implement several interfaces that enums do not declare or implement, even though their underlying types support them.
  • Some examples of the interfaces I might include are:
    • IBitwiseOperators<TSelf,T2,TSelf³>⁴
    • IComparisonOperators<TSelf,T2,bool⁵>⁴
    • IEquatable⁴⁶
    • IShiftOperators<TSelf,int,TSelf>
    • IUnaryNegationOperators<TSelf,TSelf>
    • IUnaryNegationOperators<TSelf,TEnum>⁷
    • IUtf8SpanFormattable⁸
    • IUtf8SpanParsable
    • And any that are required as a consequence of those that are implemented.
    • Many of the numeric interfaces are pretty likely to make it into the implementation, since they're commonly used and a subset of them are almost supported by enums.
• There will be at least one custom interface defined that all of the generated struct types implement, to enable easy use in situations such as generic methods and to make it easier to reuse code (though if those come up in Terminal.Gui itself, they should probably therefore get added to this generator).
  • Such interface(s) will have common items that all are guaranteed to implement, such as a subset of the above interfaces as well as easy access to useful/common values and functions.
  • If it makes sense for that to be hierarchical or composable, I might define it that way, but probably not unless I see a common and valuable use case for it.
  • Of course, anything included in such interface(s) won't be allowed to be suppressed, since that would break the interface.
• Generated structs will have boolean properties for indication of the following values/conditions:
  • All flags 0
  • All flags 1 (This does not mean the same as uint.MaxValue. It means that all defined values are 1, which enums do not have a facility for unless you make an explicit label for it.
  • Count of how many flags are set to 1
  • Count of how many flags are set to 0
  • Easy mask application and comparison
    • Could include masking via any of multiple of: TSelf, TEnum, TBacking, int, uint
    • Could potentially provide a feature for run-time and/or compile-time definition of masks in a convenient/easy way, which would eliminate the need for defining named members to cover combinations of other members. I started work on that in the other generator, but tabled it to focus a bit on getting the basic work done. Or, I could not do that and just leave it up to the fact that they're partial and you can extend it yourself when you need those kinds of members.
  • This functionality is likely (but not for sure yet) to be part of the above-mentioned interface(s).
  • Ideally, these will all also be able to have a default implementation in the interface(s), to make the interface even more flexible beyond just these types.
• Generated structs will have conversions/casts available to and from at least BOTH int and uint, with lower overhead than an enum would and with no boxing, regardless of TBacking.⁹ (they're bit vectors, remember?)
• They will have conversions/casts avaialable to and from the enum they are based on.
• They will have ToString and Parse behaviors similar (not identical and in some ways intentionally different) to enum.
• The following interfaces are likely to be implemented, IUtf8SpanParsable<TSelf>, and maybe also IUtf8SpanParsable<TEnumUnderlyingType>
• Boxing will be avoided when possible and feasible, in the generated types (enum is pretty hard, even in .net8.0, to avoid boxing with, for many common uses).
• Every member of the enum will have a corresponding boolean instance member with a name consisting of the same name but with a prefix such as Is or Has, in accordance with design guide recommendations for boolean instance properties, in the generated struct.
  • I might make an attribute for this, to allow you to override the prefix or something like that.
  • Alternatively, I could also just make that a requirement on your part and have the analyzer throw a compile error if each enum member's name doesn't start with an approved prefix.
• The actual enum members themselves (the things that feel like constants but aren't actually constants in most uses) will get corresponding ACTUAL constants of the same name, verbatim, as TBacking⁹, to make them still feel like enums when you want to use them that way (and also make it easier to switch to the new types with minimal changes).
  • Some restrictions may apply, but will be documented and reasonable if any do come up. In particular, naming is likely to just have a few reserved terms, but I may just let that be a compiler error since you can easily fix it yourself.
• They'll ideally have a fluent-style API for building values, so you don't have to use bitwise combinations to do it all. For example, that enables things like SomePimpedEnumType.Flag1.AndFlag2. I kinda like the way that's done in NUnit, where you can basically build a readable sentence and it basically builds an expression tree for you. We'll see how ambitious it's reasonable for me to get on that, though.
• They might have a couple of events and delegates (or methods taking delegates) for certain functionality or extensibility targeted at consumers of Terminal.Gui (since Terminal.Gui devs can just extend the types directly, making that unnecessary for us).

There are of course finer and further details that are open questions right now, and most of the above is also open for comment, suggestion, debate, feedback, etc. and such is welcome and encouraged

Epilogue¹⁰

So yeah... That's what I'm thinking.¹¹

Where are they now?¹²

I am currently in the design phase for a prototype struct to use as my basis for the generated code.

It is supposed to have a significant portion of the above partial spec (and currently does, though most of it is just NotImplementedExceptions, so I can compile and test things against the general structure).

That currently lives in a clean project that isn't part of our repo or solution, to ensure I am starting from a place of zero dependencies and to have a fully-working type as a model for comparison for the generator. Once I'm ready to start making the actual analyzers and generator, I'll stick that code in both the Terminal.Gui.Analyzers.Internal.Debugging and Terminal.Gui.Analyzers.Internal.Tests projects.

Ok, actually done now.

Time to do a little work and stop hurting your eyeballs.¹³

Footnotes

1. Most modern machines are little-endian, anyway, but some ARM variants can be configured as big-endian, and I just don't promise, at this time, that the generators or generated code will be cool with different byte orders. Terminal.Gui already doesn't do that anyway and nobody has complained. I'm actually not even sure if .net 8 can even be installed on big-endian machines any more, unless they can operate in little-endian compatibility modes. ↩

2. I might create a simple analyzer to check for a small set of obvious cases of bad values being assigned. There are basically infinite possibilities there, though, so I probably won't bother and will just leave it up to exceptions for a developer to fix their mistake in debugging, since that's already better than an enum or, if I do it, it'll probably only cover things like direct assignments of compile-time constants or something like that. ↩

3. TSelf means the generated type ↩

4. For these interfaces, T1 and T2 mean, in both orders (T1,T2 and T2,T1), TSelf as well as, where appropriate and feasible, possibly others like TEnum, TBacking, int, and/or uint for example. ↩ ↩² ↩³

5. A concrete type argument means all potential forms will always have this type in this position. ↩

6. Note that record types implicitly implement IEqualityOperators<TSelf,TSelf,bool>, and those operators cannot be overridden, so that will be there anyway and will be as defined by the language (value equality of all fields). ↩

7. Forms such as this are only relevant to and only possible in assignment situations, and can avoid an additional cast. T1 may include TEnum, TBacking, int, and/or uint. ↩

8. These IUTf8* interfaces bring a bunch of other interfaces with them, so those will happen as well. ↩

9. TBacking means the "underlying type" (official terminology) of the enum. That means int, by default, but could also be uint, if specified. Other types aren't planned to be supported at least initially. ↩ ↩²

10. ProEpiloge? 🤔 ↩

11. Bet you didn't think I could write something that short, huh? Wait... Does this ruin that? ↩

12. Maybe this should have been Epilogue. ↩

13. Yeah, that ruined it... Sorry 😅 ↩

[dodexahedron]

Also, for the record, I'm actually writing the core implementation of this anyway, for work projects, so I'm spending that effort no matter what. May as well let us benefit from it here, too (plus, I use TG as well, so it's at least triple-dipping and dogfooding 😅).

Point is, though, that it is my highest priority here, right now, not only because I already felt the need for it, but because it's also a priority for me outside of this project. 😃

[tig]

Do you know of another OSS project using a similar technique to success?

I love all of this, but reading the above, it feels more like an experiment than a tried-tested methodology. I'm leery of TG being a playground for fundamental science experiments.

[dodexahedron]

First of all, let me say it is great that you voice these kinds of concerns.
Please, as always, my first and most important request is for free and open communication, and I'm REALLY hard to offend. XD

But fear not! I've written another bedtime novel below!

For your first question, short answer is: "OMG very yes and have you seen---." whoops. Not short. Ok, I'll just do long like you know I will anyway....

There are some pretty decent ones from reputable people, yes. Plenty of non-free licensed and/or paid ones, as well, as people bank what they can before everyone finally starts doing it in every application of any consequence.

I actually originally wanted to use some of the stuff from this, which is one of many high-quality projects from a pretty prolific MVP (with a great technical blog, BTW), and I actually do make use of some of his stuff in the generator project already.

The enum functionality, though, didn't do as much as I wanted, and also added some build dependencies that I wanted to keep nice and clean.

Some of the conceptual basis for some of the functionality is or was learned or inspired by some of Gérald's articles and/or projects, as well (also MIT licensed, so we're all compatible), and I actually switched to his polyfill library just a day or two ago, which allows me to ditch the polyfills I had been manually writing as the needs arose. It's also a generator and therefore not a runtime dependency, which is beautiful and saved me some boring and annoying work.

At a much higher level, though, and some history, Roslyn generators have been around for almost 15 years (it's how most language features are implemented nowadays, if they don't need new binary behaviors), and were in public preview for several years until they FINALLY called it general release in .net 7. But it's THE compiler for .net and Visual studio as you know it wouldn't be nearly as powerful as it is in so many things we take for granted without it.

I've been writing analyzers and generators for various things for a long time, as well, including during the preview period. On top of that, again, it's THE compiler you've been using for quite some time (and is all on github, BTW). So, personally, I'm more than comfortable using it and find it to be one of the most powerful force multipliers I have at my disposal, for software development.

If you understand the language and you even 10% understand visual studio and msbuild, you've got all you need, aside from the usual learning specifics of an API and all that.

But yeah, even have a look on NuGet or GitHub if you like. Search for Generator or Roslyn. Tons of stuff out there, in the usual range of "wow that's amazing" to "how is this person allowed to use a computer?"

Specifically for this particular generator, there's not really anything new, per se, that isn't or wouldn't otherwise be under the general umbrella of "business as usual" for software development, for me. You know - stuff like hunting up the APIs and documentation you need, applying them, writing code, debugging that code, rinse, repeat. 🤷‍♂️

But it is something new to this specific project/organization, so I'm being extra verbose and specific, with the intent of providing as much of a 1-stop shop for others who may not have experience with it to pick it up and run with it without having to do all the research and whatnot I've done over the years. Have you seen the sheer volume of comments in the first generator, for example? I basically turned one of the core methods in it into an article all to itself, explaining what and why stuff is happening, all the way down to at least one spot where there's even mid-expression comments in a method chain. 😆

As for the verbosity of this issue post, that's also mostly for everyone else's benefit, but also trying to inject a bit of a formal spec into the process, because I've been writing software for 30 years and I long ago came to value explicit clarity and potential redundancy over brevity in that sort of thing for soooo many reasons.

But, also, it's meant as a showcase of the value and power that this sort of approach can have, when we wield the tools to more of their potential.

Plus it's a written log of thoughts and things that have been considered and whatnot.

Most importantly, it's intended to be the basis for further design talks, brainstorming, etc., even not necessarily related directly to this specific component, because there are plenty of other places we could benefit from (and that goes for pretty much every project out there).

But also, realizing that not everyone has experience with the nuts and bolts of the generators themselves, I'm putting some extra effort into making the first few big ones be as simple to use (and therefore also more likely TO be used) for everyone, without them ever having to touch Roslyn directly, if they don't want to, but still reaping the benefits (just like installing a plugin in VS or a nuget package).

But yeah... Roslyn is mature. .Net since core 3 wouldn't be what it is without it.

The flow of things in the API is really the "hardest" thing to get a handle on, IMO, but the actual concepts aren't any more difficult to grok than the language itself, because you're just writing code to...well...write code! :)

But I'm also, as I pointed out a few times in the big spec post, not designing the generators for a first year CS student who never wrote a line of code in their life. It's designed for us, first and foremost, so it's going to hold your hand in ways I may assume are helpful, but isn't going to baby you on things that are basic enough to be unacceptable in production code anyway and which unit tests can trivially catch long before that.

In (kinda/relative to above) short:

• Lots of uses out there.
• You're using it already, as is everyone else (is there a bigger example of open source than .net other than linux?).
• This already directly makes use of some high-quality Roslyn goodies from not-Microsoft.
• I've been using the nuts and bolts of it for quite a while.
• Nothing here any more experimental than any other new class being added to a codebase.
• Unit tests of the generators provide very strong guarantees of compile-time and run-time fitness.
• Might look intimidating, but it's not really.
• Even simple generators can easily create an improvement in overall code quality.
• You don't need to know anything about writing a compiler - just about writing code.
• Don't mistake verbosity for uncertainty. :)
• You will be assimilated.
• Resistance is futile.

[dodexahedron]

Or, for one of the central points, but as the "kids these days" might say it...

#NoFilter #LegitLitBasedNoCap #ITotallyShipRoslyn #SlayItGirl

[dodexahedron]

Also, as a simple matter of personal and professional ethics, I wouldn't perform or make any experimental or otherwise dangerous acts or contributions to any project, on their turf or my own, without making it unambiguously clear at the very minimum, and ensuring acknowledgement of it.

I've stuck to that my entire career, always have, always will, and wouldn't have it any other way and one of my biggest peeves is unethical behavior in any context. You can put a stamp on that. 🙂

After all... I even sign my commits. XD