[Feature request] Use surveyor to assess whether an API doc is complete or not

[jayoung-lee]

A recent analysis on the Flutter’s API doc satisfaction showed that users find API docs with full sample code (snippet) more satisfying.

This new feature may scan the class files that generate API docs and identify which class doc contains what content, so that we can tell which classes need more attention, without going through the docs manually.

For example, the Card class's dart file creates an API doc that includes: a short code sample, a full sample in an app (snippet), and an image.

We can identify these by looking at the patterns like these in the code.

1. short code sample: {@tool sample -- See TextField class for an example.
2. short code sample AND full app (snippet): {@tool snippet -- See Card class for an example.
3. image: ![image_description_here](image_url_here.png) -- This pattern may be within a sample.

If a surveyor can scan the dart files and identify the patterns, then the surveyor is expected to generate a file that looks like this.

api, sample, snippet, image // these are column names
TextField, 1, 0, 0
Card, 1, 1, 1
...

Note: There is a script that can show whether a doc has sample or snippet.
TMPDIR=/tmp dart dev/bots/analyze-sample-code.dart --temp=samples running from the flutter directory will generate outputs in /tmp/samples. However, this is not in a desirable format, and does not contain image information.

[pq]

This sounds like a great use of the tool (either cribbing from analyze-sample-code or rolling something similar of our own). @jayoung-lee: do you know if there's interest in motivating a broader survey of API docs in response to the satisfaction study?

[jayoung-lee]

I should have been more clear about the context of this request!

We have a satisfaction survey (HaTS) that's been running on the Flutter's top 20 API docs and collecting satisfaction scores since stable 1.5 release. Because the survey will continue to collect the satisfaction scores, I plan to use the surveyor to help with the analysis. More specifically, I plan to run the surveyor after each release of stable to see if any doc's been updated. Then, I will join the surveyor results with the HaTS survey results, which will help me more easily track how the changes in the API docs (e.g., added image) influence the satisfaction (instead of going through 20 pages manually after each release).

What I can also do is creating a dashboard that shows the ranking of API docs by the number of visitors (from Google Analytics) along with the information on whether a doc is missing snippets or images (surveyor output), so that the team members can easily identify which doc needs love.

Another use case that @InMatrix suggested is to use the same surveyor for the documentation of Flutter/Dart packages. From the Q2 survey we found out that one of the biggest pain points in using packages was the lack of documentation, especially the code samples (article). I think we can propose to use the surveyor outputs as one of the quality metrics.