Reimplement the extension annotation processor

[gsmet]

Warning

This is a work in progress, do not merge. You can try it at home though.

Fixes #35439

This is the current status for the new extension annotation processor.
If you want to test it, make sure you install it separately first as it looks like Maven doesn't order things as it should. That's something on my TODO list.

I kept the old code as I thought it could still be working but I have some issues I haven't figured out yet. So a full build of the docs will fail in the documentation module for now.

You can fully ignore the generate_doc package as it's the old stuff that will be dropped.

Main changes

• Separation of concern between what we need for the extension build and what we need for the documentation
• Proper scanning phase that goes through the annotations followed by the build of a model. The model is agnostic of what we will want to do with it: we will be able to use it to generate the doc but also to generate files for IDEs
• Separate implementation for @ConfigMapping and legacy @ConfigRoot to avoid having a gazillion of hacks and corner cases
• The current annotation processor writes files in the root target/ and doesn't respect the boundaries of the module: this prevents us from caching the compiler goal for the extensions. The reason for it is that you can have config for the same extension in the build time and runtime modules, you can have a config group that is in a shared module... which makes it impossible to fully resolved the documentation at the module level. The main idea here is to generate a model per module and then merge the model at the end (in some cases, the merge will be easy, in some others, it's going to be harder - for instance when we were not able to resolve a config group). I'm still experimenting with this part and have several options in mind. That's my next step.
• We don't write a gazillion of temporary files anymore: while the annotation processor works in rounds, it is initialized only once so we can keep the status in plain fields.
• A Maven plugin does the final assembly and generate the Asciidoc content

Other changes

• You cannot mix @ConfigMapping and legacy @ConfigRoot in the same extension anymore. It used to not be reliable at all as the detection was quite broken. While I think we could make it work more reliably with the new code, I think we should keep it strict to enforce more checks.
• The annotation processor is only executed for the main compiler execution, not for the tests

Current status

• The scanner is working fine
• We can build a model for modules - I tested it with some non trivial modules such as Hibernate ORM and Hibernate Search and the model was in line with what we would expect
• The build is passing (except I broke the legacy doc so the documentation module doesn't build fine - not sure if it's a simple breakage or if this is inherently broken)
• Some modules mix @ConfigMapping and legacy @ConfigRoot: Core module is using a mix of traditional @ConfigRoot and @ConfigMapping #42114 - for these two extensions, I specifically support a mix of both.
• I added a Maven plugin to generate the Asciidoc (still a WIP) - it uses Qute.

What's missing

- I need to work on the final assembly of the model: I have two strategies to experiment with - one should be very simple - but is not what I experimented with here, and the other one is more complex and is what I started to implement here (I had the former idea today but not yet sure if it can work).
- Once the final assembly is done, we will need to generate the config doc from the module. Atm, I'm thinking that it would actually be good idea to use a JBang Quarkus script and some Qute templating, rather than a Maven plugin. Still a bit unsure as we also need to make sure this is easily usable by extensions in the Quarkiverse and versioned. One thing is for sure, I'd like to use templates as the current generation in plain Java is really hard to follow. @aloubyansky @mkouba maybe you have some thoughts on this?

• Still some work to do on the Maven plugin generating the AsciiDoc
• A lot of manual checks to compare the new output to the current one
• Ideally, we would need to write some tests, not entirely sure yet how we will proceed with that.
• Once this is in, we will be able to enable the cache for the compiler goals of the extensions, which should speed up the build but we will need some tweaks to https://github.com/quarkusio/quarkus-project-develocity-extension first.

How to experiment

If you want to experiment, here are a few steps:

• get the code of this PR
• install the annotation processor with mvn clean install -Dquickly -f core/processor
• cd extensions/your extension of choice
• mvn clean install -DskipTests
• in the runtime and deployment modules, in target/quarkus-config-doc, you should have a YAML file containing the resolved model: you can have a look at what you have here, it should be in line with what's currently documented. If not, please point it to me and I will have a look. Note that modules that reference config groups or superclass/interfaces that are outside of the module boundaries will not be fully resolved but for all the simple extensions out there, it should work fine.
• once you are done with your experiments, make sure you install a vanilla annotation processor from main with mvn clean install -Dquickly -f core/processor

Model looks like:

---
configRoots:
  quarkus.hibernate-validator:
    extension:
      groupId: "io.quarkus"
      artifactId: "quarkus-hibernate-validator"
      name: "Quarkus - Hibernate Validator"
    prefix: "quarkus.hibernate-validator"
    items:
    - !<io.quarkus.annotation.processor.documentation.config.model.ConfigProperty>
      sourceClass: "io.quarkus.hibernate.validator.runtime.HibernateValidatorBuildTimeConfig"
      sourceName: "failFast"
      path: "quarkus.hibernate-validator.fail-fast"
      type: "boolean"
      description: "Enable the fail fast mode. When fail fast is enabled the validation\
        \ will stop on the first constraint violation detected."
      phase: "BUILD_AND_RUN_TIME_FIXED"
      environmentVariable: "QUARKUS_HIBERNATE_VALIDATOR_FAIL_FAST"
      typeDescription: "boolean"
      defaultValue: "false"
    - !<io.quarkus.annotation.processor.documentation.config.model.ConfigSection>
      sourceClass: "io.quarkus.hibernate.validator.runtime.HibernateValidatorBuildTimeConfig.HibernateValidatorMethodBuildTimeConfig"
      sourceName: "methodValidation"
      path: "quarkus.hibernate-validator.method-validation"
      type: "io.quarkus.hibernate.validator.runtime.HibernateValidatorBuildTimeConfig.HibernateValidatorMethodBuildTimeConfig"
      title: "Method validation"
      items:
      - !<io.quarkus.annotation.processor.documentation.config.model.ConfigProperty>
        sourceClass: "io.quarkus.hibernate.validator.runtime.HibernateValidatorBuildTimeConfig.HibernateValidatorMethodBuildTimeConfig"
        sourceName: "allowOverridingParameterConstraints"
        path: "quarkus.hibernate-validator.method-validation.allow-overriding-parameter-constraints"
        type: "boolean"
        description: "Define whether overriding methods that override constraints\
          \ should throw a `ConstraintDefinitionException`. The default value is `false`,\
          \ i.e. do not allow.\n\nSee Section 4.5.5 of the JSR 380 specification,\
          \ specifically\n\n[quote]\n____\nIn sub types (be it sub classes/interfaces\
          \ or interface implementations), no parameter constraints may be declared\
          \ on overridden or implemented methods, nor may parameters be marked for\
          \ cascaded validation. This would pose a strengthening of preconditions\
          \ to be fulfilled by the caller.\n____"
        phase: "BUILD_AND_RUN_TIME_FIXED"
        environmentVariable: "QUARKUS_HIBERNATE_VALIDATOR_METHOD_VALIDATION_ALLOW_OVERRIDING_PARAMETER_CONSTRAINTS"
        typeDescription: "boolean"
        defaultValue: "false"

[...]

[aloubyansky]

@gsmet could you please check the docs build failure?

[gsmet]

Ah yes, last minute change...

[maxandersen]

I'm going to try check myself today but just asking the question here if its something you already considered:

is the annotation processor model capturing the "open-ended" properties in a way that tools (IDE) can reason about it?
i.e. like discussed/showed in https://groups.google.com/g/quarkus-dev/c/JXvndoAsekQ/m/YD4K0LYaAwAJ for kubernetes the IDE tools store kubernetes.init-containers[*].image to know that it allows for either no name default or a named section.

similar for jdbc datasources.

I assume it does; but didn't spot it in the example shown.

[gsmet]

@maxandersen I really want to get this in as I want to work on follow ups after that without having the risk of conflicts.

We can tweak things later if you have comments.

As for your question: yes we have all the information necessary to generate whatever we want for the IDEs, and Maps are supported as they are a key element of our config.

A good example of that is what we end up generating for Hibernate ORM Elasticsearch:

quarkus.hibernate-search-orm.elasticsearch.schema-management.mapping-file
quarkus.hibernate-search-orm.elasticsearch."backend-name".schema-management.mapping-file
quarkus.hibernate-search-orm."persistence-unit-name".elasticsearch."backend-name".schema-management.mapping-file
quarkus.hibernate-search-orm."persistence-unit-name".elasticsearch.schema-management.mapping-file

The format is not exactly the same but as you can see we have all the information to indicate we have a Map.

This was always supported except it was hard to exploit the data.

For the IDE format, we will just have to define the format we want and we will generate something they can understand as we have the proper data structures to do it (and they are easy to tweak if something is missing but I think we have all we need).

[quarkus-bot]

Status for workflow Quarkus Documentation CI

This is the status report for running Quarkus Documentation CI on commit 083795b.

✅ The latest workflow run for the pull request has completed successfully.

It should be safe to merge provided you have a look at the other checks in the summary.

Warning

There are other workflow runs running, you probably need to wait for their status before merging.

[maxandersen]

cool @gsmet - to be clear I'm not saying no to merge in any form. this As far as I see this doesn't change or introduce new things it cleans up the process and allows (in future) to have config doc generation much saner and scalable.

I've wanted this stuff for ages; just looking to spot corner cases.

I generally just don't want to put a +1 LGTM on a big PR before having had a chance to run it :)

Thus, if you and @aloubyansky are confident about carrying it into the 3.14/3.15 release, go for it.

[gsmet]

Let's say I'm confident enough that the new stuff is a lot more flexible and we will be able to achieve what we want to do.

In September, I'll start the discussion with the various tooling teams to define what they are interested in.

[quarkus-bot]

Status for workflow Quarkus CI

This is the status report for running Quarkus CI on commit 083795b.

✅ The latest workflow run for the pull request has completed successfully.

It should be safe to merge provided you have a look at the other checks in the summary.

You can consult the Develocity build scans.

[github-actions]

🙈 The PR is closed and the preview is expired.

[maxandersen]

It was just that the diff grew from 520ish to 620ish files thus trying to spot what caused the 100 files extra change. If change of includes then makes sense. /max

…

On 7 Aug 2024 at 11:49 +0200, Guillaume Smet ***@***.***>, wrote: @gsmet commented on this pull request. In docs/src/main/asciidoc/cache-infinispan-reference.adoc: > @@ -209,4 +209,4 @@ the cache. quarkus.cache.infinispan.my-cache.max-idle=100s ---- What do you mean, they weren't there earlier? I'm updating existing includes all over the place? Or can you point me to the place where I add some? The reason why the naming is different is that the annotation processor is a lot smarter now and the naming used to be derived from the config root name with a lot of weird corner cases. The naming is now a lot more consistent. — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you were mentioned.Message ID: ***@***.***>

[maxandersen]

It was just that the diff grew from 520ish to 620ish files thus trying to spot what caused the 100 files extra change. If change of includes then makes sense. /max

that was sent 24 hrs ago via email....i guess github had struggles :/