Elements or Lower

Thu, 18 May 2006

SiteMorse Testing

Right now, there’s a fascinating discussion in progress over at Accessify Forum on the merits and problems with the SiteMorse automated site testing tool, specifically insofar as it claims to test pages against the WCAG.

Regular readers here will know that SiteMorse publish a monthly league table of both central and local government sites, and that despite these having no formal government backing, it’s hard to ignore them altogether. The league table divides its results into a number of columns, including “performance”, “metadata”, “code quality”, “accessibility” and “function”. We’d been consistently scoring well on most of these, but very poorly on “function”, and we had no real idea what precisely this meant, nor did we get very far in finding out by trying to contact SiteMorse directly.

So, we decided to purchase a bunch of credits for the SiteMorse tool, in order to fully divine what they perceived was wrong with the site, and to be able to make better sense out of the monthly league table. In the light of the current discussion, I thought it might be useful to summarise what we found.

For the record, it turned out that “function” relates to things like broken links (both internal and external) or malformed URLs, together with a handful of potential issues in the HTML itself, such as lacking a <title> element. It also turned out that any problems in this area, such as a link to an external site timing out, would be marked quite punitively. With an appropriate login, the full results from the league table tests are made available, and the most recent test (carried out on 8 May, and covering 12 pages from the site) had a link to each of two different sites time out. These two errors alone furnished us with a function score of 2/10 — ouch!

Of course, part of the reason for this is that the SiteMorse tool simply balances the number of errors found against the number of pages tested. If your site uses correct syntax, but does nothing, has no content, and links nowhere, your SiteMorse score will be fantastic. A great SiteMorse score isn’t really an endorsement of what you’re doing, but merely an indication that you managed to avoid a specific set of potential errors.

This has a great impact on the interpretation of the SiteMorse accessibility score. A SiteMorse accessibility score of 10/10 does not equal a fully-accessible page, but merely indicates that there were no failures discovered for any of the 16 different accessibility checks it actually carries out. Of course, it’s very hard to keep this in mind, even though the page for accessibility in a given test lists a further 26 points with “manual check” written against them. What’s more, there seems to be a very strong disconnect between the disclaimers in the SiteMorse test results themselves, and the tone of the company’s marketing.

Despite all this, we found the accessibility tests genuinely useful in identifying a handful of real issues. We had a few pages, adapted from old CGI scripts, that contained deprecated HTML. We had a few pages with out-of-sequence headings. We had a couple of pages whose content didn’t validate properly. And we had a truckload of links which used the same link text, but with different link URLs.

In addressing these issues, it was really important to be able to interpret the results in a sensible and educated way. For example, just as one could avoid broken link errors by having no links, one could avoid heading-sequence errors by having no headings. I don’t think it’s entirely fair to consider this an argument against the tests themselves, however. It’s more of a caution not to prioritise a “good” result over a good site, league tables be damned.

Incidentally, we noticed that the links with identical link text but different link URLs were, without exception, internal links. Because the CMS handles the generation of the HTML for internal links, it was relatively straightforward to adapt this to include sensible disambiguation in a title attribute for each link. To see some examples of this, view source on the sitemap.