Bruce Lawson’s personal site

Review: Evinced accessibility site scanner

(Last Updated on )

That nice Marcy Sutton asked me to test and give feedback about a new product she’s involved with called Evinced, which describes itself as an “Enterprise grade digital accessibility platform for modern software development teams”. Quite what “enterprise grade” means is beyond me, but it’s basically software that can crawl a website from a root domain and check its code against some rules and report back. There are similar tools on the market, and I’ve recently been working with a Client to integrate Tenon into their workflow, so wanted to compare them.

“Now hang on!”, I hear you say. “automated tools are terrible!”. Well, yes and no. Certainly, overlays etc that claim to automatically fix the problems are terrible, but tools that help you identify potential problems can be very useful.

It’s also true that automated tools can’t spot every single accessibility error; they can tell you if an image is missing alternate text, but not that <img src=dog.png alt=”a cat”> has useless alt text. Only a human can find all the errors.

However, many errors are machine-findable. The most-common errors on WebAIM’s survey of the top million homepages are low contrast text, missing alternative text, empty links, missing form input labels, empty buttons and missing document language, all of which were found by running automated tests on them (which, presumably, the developers never did before they were pushed to production).

I personally feel that a good automated scanner is a worthwhile investment for any large site to catch the “lowest hanging fruit”. While some things can’t be automatically tested, other things can, and other aspects live in a grey area depending on the rigour of the test.

For example, a naive colour contrast test might compare CSS color with background-color, and give a pass or fail; a less naive test will factor in any CSS opacity set on the text and ignore off-screen/ hidden text. A sophisticated contrast test could take a screenshot of text over an image or gradient and do a pixel-by-pixel analysis. To do a test on a screenshot would require actually rendering a page. Like Tenon, Evinced doesn’t just scan the HTML, but renders the pages in a headless browser, which allows the DOM to be tested (although I don’t believe it tests colour contrast in this way).

Evinced uses Axe Core, and open-source library also used by Google Lighthouse. It also contains other (presumably proprietary secret-source) tests so that “if interactable components are built with divs, spans, or images – Evinced will detect if they are broken”.

Reporting

The proof of the pudding with automated site scanners is how well they report errors they’ve found. It’s all very well reporting umpty-squllion errors, but if it’s not reported in any actionable way, it’s not helpful.

Like all the automated scanners I’ve tried, errors are grouped according to severity. However, if those categories correspond with WCAG A, AA and AAA violations, that’s not made clear anywhere.

Graph showing number of errors by type

It’s a fact of corporate life that most organisations will attempt to claim AA compliance, so need to know the errors by WCAG compliance.

One innovative and useful reporting method is by what Evinced calls component grouping: “Consolidates hundreds of issues to a handful of broken code components”.

With other scanners, it takes a trained eye to look through thousands of site-wide errors and realise that a good percentage of them are because of one dodgy piece of code that is repeated on every page. Evinced analyses pages and identifies these repeated components for you, so you know where to concentrate your efforts to get gross numbers down. (We all know that in the corporate world, a quick fix that reduces 10,000 errors to 5,000 errors buys you time to concentrate on the really gnarly remaining problems.)

graph showing 32% of all issues are grouped into 38 components; 1 component accounts for 81% of critical issues

There’s a vague suggestion that this grouping is done by Artificial Intelligence/ Machine Learning. The algorithm obviously has quite clever rules, and shows me a screenshot of areas on my pages it has identified as components. It’s unclear whether this is real Machine Learning, eg whether it will improve as its corpus of naughty pages gets larger.

list of components automatically identified on my site with screenshots and the relevant areas highlighted

I don’t recall signing anything to allow my data to be used to improve the corpus; perhaps a discount for not-for-profits/ time-limited demos could be offered to organisations allowing their data to be added to the training data, if indeed that’s how it “learns”.

User Interface

Many of these site scanners are made by engineers for engineers and have the similar high levels of UX one would expect from JavaScripters.

Tenon has some clunkers in its web interface (for example, it’s hard to re-run a previously defined scan) because it’s most commonly accessed via its API rather than web back-end.

Evinced makes it easy to re-run a scan from the web interface, and also promises an API (pricing is not announced yet) but also suffers from its UI. For example, one of my pet peeves is pages telling me I have errors but not letting me easily click to see them, requiring me to hunt. The only link in this error page, for example, goes to the “knowledge base” that describes the generic error, but not to a list of scanned pages containing the error.

Page showing how many errors I have but not linking to them

(After I gave feedback to the developers, they told me the info is there if you go to the components view. But that requires me to learn and remember that. Don’t make me think!)

There’s also terminology oddities. When setting up a new site for testing, the web interface requires a seed URL and to press a button marked “start mapping”, after which the term “mapping” is no longer used, and I’m told the system is “crawling”. Once the crawl was complete, I couldn’t see any results. It took a while for me to realise that “crawling” and “mapping” are the same thing (getting a list of candidate URLs) and after the mapping/ crawling stage, I need to then do a “scan”.

A major flaw is the ability to customise tests. In Tenon I can turn off tests on an adhoc basis if, for example, one particular test is giving me false positives, or I only want to test for level A failures. This is unavailable in Evinced’s web interface.

Another important but often-overlooked UI aspect of these “Enterprise” site scanners is the need to share results across the enterprise. While it’s common to sell “per-seat” licenses, it’s also necessary for the licensee to able to share information with managers, bean-counters, legal eagles and the like. Downloading a CSV doesn’t really help; it’s much more useful to be able to share a link to the results of a run and let the recipient investigate the reports and issues, but not let them change any configuration or kick off any new scans. This is missing in both Evinced and Tenon.

Conclusion

The system is currently in Beta and definitely needs some proper usability testing with real target users and UI love. One niggle is the inaccuracy of its knowledge base (linked from the error reports). For example, about the footer element, Evinced says

Since the <footer> element includes information about the entire document, it should be at the document level (e.g., it should not be a child element of another landmark element). There should be no more than one ><footer> element on the same document

This is directly contradicted by the HTML specification, which says

The footer element represents a footer for its nearest ancestor sectioning content or sectioning root element… Here is a page with two footers, one at the top and one at the bottom… Here is an example which shows the footer element being used both for a site-wide footer and for a section footer.

I saw no evidence of this incorrect assumption about the footer element in the tests, however.

All in all, the ability of Evinced to identify repeated ‘components’ and understand the intended use of some splodge of JavaScriptted labyrinth of divs is a welcome feature and its main selling point. It’s definitely one to watch when the UX is sleeker (presumably when it comes out of Beta).

One Response to “ Review: Evinced accessibility site scanner ”

Comment by Illai Zeevi

Hi Bruce
First of all we want to thank you and say that we are very appreciative that you took the time to test our product and write about it in your blog.
Regarding your comment about the inaccuracies in our knowledge base we would like to make a clarification. The entry you referred to is under the “landmarks” section, footer elements are considered as landmarks only when they are on the root level or when role=”contentinfo” is used. In these cases the must be unique according to the WAI-ARIA specification.
However, we understand why the way this entry was written might look confusing and not accurate. We rewrote this entry to be more clear and it is already updated on our knowledge base.

Thanks
Evinced

Leave a Reply

HTML: You can use these tags: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong> . To display code, manually escape it.