Step Analysis / Code Generation

[Code-Grump]

Because I like to juggle multiple parallel things, and inspired by similar static analysis available for Microsoft's logging system, I thought I would look into adding a Roslyn analyser for Reqnroll steps. Specifically, something that would look at the attributes assigned to a step and verify:

1. The syntax of the text is complete and will compile to a regular expression or Cucumber expression.
2. That the number of matching groups / parameters in the expression matches the number of arguments to be passed to the step method

But then I also got thinking: why are we compiling step bindings at runtime when we could be doing that at compile-time? This is how the new compile-time Regular Expression support works, turning a partial method decorated with an attribute into a complete implementation of a regular expression. Likewise, we could turn every step binding attribute into a compiled member that can match and invoke the step.

This means we'd have:

• No need for runtime reflection to invoke steps (because we can generate all that code at runtime)
• No need for parsing step bindings during execution (because we can do all parsing at compile-time)
• Trivial step discovery (because we can generate and expose a collection of them for every assembly)
• An opportunity to pre-compile step-text matching for even faster execution.

There's overlap here; any validation performed by a static analyzer would be somewhat redundant if a code-generator was also going to repeat that work, but both could exist side-by-side.

What would it take to achieve an analyzer?

• Create a new model for parsing step binding text which exposes the metadata needed for parameter analysis. This would have to account for both expression languages we support, ideally expressed as a single model.
• Load that parser into a Roslyn analyzer to indicate step binding syntax errors.

What would it take to achieve code-generation?

• Same new model for parsing step bindings as analyzer
• A code generator to turn step attributes into a registry of generated step classes
• Generalising the runtime to allow runtime-bound and compiled steps to be invoked side-by-side
• An extension to the runtime's discovery of steps to look for compile-time generated step registries exposed by assemblies

Does this seem useful? I certainly like moving runtime concerns to compile-time concerns wherever possible and this seems like a pretty good candidate. The only thing that might make it undesirable is if we eventually wanted to abandon step discovery at runtime completely - make it so we can only bind to steps that are discoverable at runtime so that feature code generation would match feature steps to step definitions at compile-time and add direct invocation. There's a lot of drawbacks and challenges to doing so (can't use dynamically loaded assemblies, data-sources have to be available at compile-time, etc), but I know the idea has been brought up as potentially being a long-term goal.

[gasparnagy]

Yes, this is certainly a good idea. I was also thinking about this as a long term goal.

There are (and probably there will be) cases where dynamic binding would be required, but still, I think vast majority of the step-calls are static. When I was thinking about this, my imagined solution was to generate a code that first verifies if dynamic binding is needed (this can be a quick check, for most of the projects as simple as "return false") and if not, than move on with the statically generated stuff.

Nevertheless I am not sure I fully understand the technical solution you summarized. Could you maybe demonstrate this with some simple pseudo-code (ie. what the user needs to do and what do we generate compile time) though a concrete example?

[Code-Grump]

The idea is to generate code that is tailored for each defined step, doing work at compilation-time that would otherwise have to happen at runtime.

[gasparnagy]

I really like this concept.

As I mentioned above, this approach does not necessarily incompatible with dynamically loaded/configured step definitions. Those dynamic ones would be stored in a different (runtime) list and the execution could first check the step against the dynamic ones and fall back to the generated ones or something similar. The matching is anyway a regex match so for that it does not matter if the code is generated or not. The problem is more with the invocation by reflection that could be still avoided by the non-dynamic ones.

My idea in addition to this would be the following:

During compilation we could try to match all known step definitions for each step and generate the invocation code to the test method itself. So using your example, let's assume a scenario like this:

Scenario: A
When I do the thing to "foo"
Then the thing is done

Then the test method we generate could be like:

public async Task ScenarioA() {
  // ... setup stuff

  // code for When I do the thing to "foo"
  if (!handle-by-dynamic-step-defs("I do the thing to \"foo\"")) {
    // if there are no dynamic step defs of they do not match:
    var stepMatch = AwesomeStepsDefinitions.WhenDoTheThing.MatchStep("I do the thing to \"foo\"");
    if (stepMatch.Success) {
      await AwesomeStepsDefinitions.WhenDoTheThing.InvokeStep(container, stepMatch.Args);
    }
    else throw...
  }

  // code for Then the thing is done
  if (!handle-by-dynamic-step-defs("Then the thing is done")) {
    // if there are no dynamic step defs of they do not match:
    var stepMatch = AwesomeStepsDefinitions.ThenTheThingIsDone.MatchStep("Then the thing is done");
    if (stepMatch.Success) {
      await AwesomeStepsDefinitions.ThenTheThingIsDone.InvokeStep(container, stepMatch.Args);
    }
    else throw...
  }
}

The benefit of this would be that if you then use the "find usages" on the step def method, it would list the feature file lines. (For that we would need to add some line pragmas to the generated step def classes, but we could make it work.) With that the code lens usage list might work as well automatically. In the worse case we could just generate an additional unused ref to the method to convince static code analysis: await AwesomeStepsDefinitions.WhenDoTheThing.InvokeStep(container, stepMatch.Args, () => AwesomeStepsDefinitions.WhenIDoTheThingTo(null) /* this is not used actually */);

Of course when we know that there are multiple possible step defs might match (e.g. with scenario outline steps), the code would be a bit more complex, because we would need to invoke all matching (to detect ambiguity) and then invoke the winning one, but something like this would work:

  // code for When I do the thing to "foo"
  if (!handle-by-dynamic-step-defs("I do the thing to \"foo\"")) {
    // if there are no dynamic step defs of they do not match:
    var multiStepMatch = HelperClass.MatchStepDefinitions(AwesomeStepsDefinitions.WhenDoTheThing, OtherAwesomeStepsDefinitions.WhenDoTheThing, "I do the thing to \"foo\""); // this would call the match on each and fail if no match or ambiguous 
    switch (multiStepMatch.MatchedIndex) {
      case 0: 
        await AwesomeStepsDefinitions.WhenDoTheThing.InvokeStep(container, multiStepMatch.Args);
        break;
      case 1: 
        await OtherAwesomeStepsDefinitions.WhenDoTheThing.InvokeStep(container, multiStepMatch.Args);
        break;
    }
  }

The HelperClass.MatchStepDefinitions could also do the dynamic step matching as well of course, so this could be the way to work even if there is only one known matching step def.

[Code-Grump]

Just to be clear: this proposes a change to the runtime step-binding such that there's no "parsing" of step definitions; they come "pre-baked" into types and registered in a convenient location so that the runtime just looks for a match and then runs the matching step.

This does pave the way for an easier time doing compile-time binding, such that the invocation of a step is just a trivial method-call. I would expect no parsing to be necessary at runtime in these cases, where we have already parsed and extracted parameters at compile-time (save for any custom transformations.)

[gasparnagy]

@Code-Grump Yes, I got that. Did I say anything that could be misunderstood and contradicts to this? (For the language clarification, I tend to use the word "parsing" for Gherkin parsing and "binding discovery" or just "discovery" for analyzing the step definition methods and attributes. This latter is what we can avoid doing runtime with this idea.)

[Code-Grump]

I would expect the call chain in TestExecutionEngine.ExecuteStepAsync to be changed so its search and subsequent step execution are modified to look at the generated step registries to match and then invoke the steps. I would expect this to be controlled by a feature-switch that is all-in: it only looks for generated step definitions or it only performs runtime binding discovery, not a mix of the two.

The generated test-fixtures wouldn't need to change - they already supply everything to perform dynamic invocation. Compile-time binding definitely gets easier on top of this model.