2021 update of the accessibility audit

[DiedeGu]

Dear reader / readers,

I have started with repeating your audit. Currently I have fully tested the tools below. There are however a few remarks on how I tested, that you may want to consider.

ARC toolkit
AXE
ASLint
HTML code sniffer
Wave
Siteimprove
Google ADT
Nu Html Checker
Enabler

Firstly, I have added 'Enabler' and 'ARC toolkit' to the set of tools that were tested. My reason was simple, I just want to test a lot of tools. Enabler is not such a well known package, but it runs in CLI only and tests pages directly, so I wanted to see what that would deliver. This might deviate from your goal of only testing a small and more prominent set of tools.

Secondly, in repeating the tests I might have weighted the same outputs differently than you have. In most tools I could not see a clear difference between warnings and errors. This is why I decided to judge all results more on the text results and how well they were able to point out a single element as the source of their problem. In some cases, for instance ASLint spotting information only given with colour, I have judged it an error although it is labelled as a warning. The indication of the problem was so spot on that I cant really see a way you can be more on point of finding this specific type of problem.

Thirdly, HTML Code Sniffer currently has a high score on giving warnings but this is a bit of a facade. They currently include 42 standard manual checks on AAA level. Since these manual checks are also more blanket statements on what to watch for, they in theory give the most suggestions on what to watch out for. In practice I think the statements will mostly go ignored because its just a lot of checks to work through every page.

Fourthly, I forgot to do the manual checks on Google's ADT, so these results will need another update.

[thibaudcolas]

Massive kudos to you for taking this on! It would be amazing for a resource like this to be kept up-to-date, although I imagine it’s a big amount of work. If you’re interested in widening the tools used, there are at least two relatively high-profile accessibility checkers which have come out since the first audit was done: IBM’s Equal Access, and Alfa from Siteimprove.

[DiedeGu]

Hi Thibaud,

Thank you! These are two checkers I did not yet find in any of the lists I encountered. I will include them. Keeping this resource up to date is something I still consider. The two main challenges are how far the current setup can be automated (I want to rerun all checks for a tool within a pipeline on any released update of a tool). And the challenge of having dynamic content problems introduced.

For now I used the project to rerun the tests because I wanted the results for a blogpost for my company, so a full set of results will eventually find its way to the pull requests here.

Thank you again for the checkers!

[selfthinker]

Many thanks @DiedeGu for reviving this.

The way the changes are done are very difficult to review because everything is in one single commit. Could you make one commit per tool? (Or even one PR per tool could make sense.) It would also be good to separate all your whitespace changes into another commit. You could either leave out generating the static files or make that another separate commit at the end.

As to adding new tools, all of those make totally sense.

Regarding your other suggestions, I will look at those later when it's easier to review the changes.

[selfthinker]

I forgot to do the manual checks on Google's ADT

Their last update was on 14 Sep 2016. So there is no need to do any checks.
(The same is true for EIII and AChecker. They had their last updates in 2015 and 2013 respectively.)

[selfthinker]

Could you make one commit per tool? [...] It would also be good to separate all your whitespace changes into another commit.

Let us know if this is something you cannot do or don't feel comfortable doing. Then we could do it instead.

[DiedeGu]

Hi @selfthinker, sorry for the wait you commented on the day I fully reinstalled my system so I have been a bit of the radar. I think I can redo the pull requests as requested.

However I would like to make sure I do them right this time. Would it be workable if I only update the tests.json for the already added tools ? This would make it all a bit more manageable. I'll give you an update when I have added them.

for the whitespaces, my editor just does those automatically. I suggest we leave them out of the commits unless you would like them in. Its no problem for me either way.

Kind Regards

Diede

PS: good to know we do not need to update Elll and AChecker :).

[DiedeGu]

Hi @selfthinker,

I could start the recommits today, did you have time to review my last comment?

Kind Regards,

[selfthinker]

Would it be workable if I only update the tests.json for the already added tools ?

Yes, sure. That sounds good.

for the whitespaces, my editor just does those automatically

Just a tip: You could install an EditorConfig plugin for your editor. That would make sure it adheres to each project's specific whitespace settings.

[selfthinker]

@DiedeGu, just to let you know that I have seen your other PRs. Thanks for doing that. Unfortunately I don't have a lot of time at the moment, so it will take a while before I can look at them.