Option to output Markdown warnings as JSON

[robertoaloi]

ex_doc uses the EarMark parser to parse Markdown embedded in Erlang files.

In case of warnings:

-module(test).
-moduledoc """
This file`s Markdown is broken
""".

ex_doc will print them in human-readable format:

mkdir test_ebin
erlc +debug_info -o test_ebin test.erl
ex_doc -f html foobar 1.0.0 test_ebin
    warning: Closing unclosed backquotes ` at end of input
    │
  3 │ This file`s Markdown is broken
    │ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    │
    └─ test.erl:3: (file)

View "html" docs at "doc/index.html"

To use ex_doc programmatically (e.g. in CI), it would be convenient to report warnings in JSON format. For example:

{
  "path": "path/to/test.erl",
  "line": 3,
  "char": 42,
  "severity": "warning",
  "name": "ex_doc",
  "description": "Closing unclosed backquotes ` at end of input",
  "original": "This file`s Markdown is broken",
  "replacement": "This file's Markdown is broken" (maybe one day)
}

[viniciusmuller]

Hey there! I'm working on this issue and I would like to make sure my understanding of the issue is correct:
We want to add a new flag that will make ex_doc only print warnings as JSON instead of the human readable format?

I've already started working on it locally and I got this example output when running the tests:

%{
  "line" => 14,
  "message" => "documentation references function \"Warnings.bar/0\" but it is undefined or private",
  "module" => "Elixir.Warnings",
  "name" => "ex_doc",
  "path" => "test/fixtures/warnings.ex",
  "severity" => "warning"
}
%{
  "line" => 19,
  "message" => "documentation references function \"Warnings.bar/0\" but it is undefined or private",
  "module" => "Elixir.Warnings",
  "name" => "ex_doc",
  "path" => "test/fixtures/warnings.ex",
  "severity" => "warning"
}
%{
  "line" => 3,
  "message" => "documentation references function \"Warnings.bar/0\" but it is undefined or private",
  "module" => "Elixir.Warnings",
  "name" => "ex_doc",
  "path" => "test/fixtures/warnings.ex",
  "severity" => "warning"
}

Could you help us by clarifying which CI systems and applications can benefit from this feature and what would be the best format to output so we can maximize effectiveness across different CI systems and tools?

[josevalim]

Ping @robertoaloi!

[robertoaloi]

Thanks for the ping @josevalim! Sorry @viniciusmuller, I totally missed the reply!

The general idea is to have ExDoc emitting issues in a parsable format, that could easily be picked by an external tool to create, for example, annotations/comments at a precise location. These annotations could be displayed on a Pull Request, or in the IDE, and even have an auto-fix available.

Internally, we use this specific format, but I don't think it's officially standardized and it's not particularly expressive (e.g. it can include an auto-fix, but limited to a string replacement - no multi-line or complex fixes).

A more standardized (but also complex) format seems to be Sarif, but that may be overkill.

Looking at how GitHub handles this, it seems like it's possible to emit annotations with this syntax. Actions such as this one exist to run a tool (generally a linter or formatter) and produce the corresponding annotations. There seems to be the concept of a matcher which can produce annotations using a basic regex extraction.

It would be important for the tool to emit an appropriate status code in case of failure (or warning, if configured as warnings-as-errors).

Looping in @michalmuskala in case he has further insights.