Drupal evaluation in Idealware CMS Survey

Here are some links to some FYI, background, and context readings that may be of interest.

Evaluating Open Source Systems is Hard:
http://idealware.org/blog/2009/03/evaluating-open-source-systems-hard.html
When Idealware was preparing to release their 2009 version of this report, Laura Quinn wrote this post describing some of the challenges involved with evaluating open source systems. This statement appears to reflect a willingness to re-evaluate or discuss the evaluations.

2009 Idealware Comparing Open Source CMS Report:
http://www.scribd.com/doc/15436026/Idealware-Comparing-Open-Source-CMS-R...
I can't locate the 2009 version on the idealware site, but here is a link to a copy of it on Scribd. I don't see any details relating to accessibility/standards at all, so it is quite possible that the 2010 report is the first time they've attempted to includes such ratings.

Choosing an Accessible CMS (Juicy Studio):
http://juicystudio.com/article/choosing-an-accessible-cms.php
This 2007 report influenced my organization's choice to go with Drupal over Joomla based on the degree of compliance with code standards and with web accessibility guidelines. And for several years after, I think this was pretty much the only publicly available report of its kind done by someone who actually knew something (quite a lot actually) about accessibility. If this 2007 report ends up being replaced by the 2010 Idealware analysis, then it will represent not just a disappointing "status quo", but will actually result in a significant drop in Drupal's current rating with respect to web accessibility as far as easily available, supposedly unbiased, (mostly) public reports on the web go.

Phil.

Had I known that there might be a concerted effort to respond, I definitely wouldn't have sent this, but since Laura Quinn took time to compose a helpful reply, I thought it best to share. (She gave permission to post the full text.)

My email:

Hi there,

There's been a bit of exchange in the Drupal Usergroup mailing list in Toronto, and I think you'll find that many informed accessibility advocates are going to find fault with your methodology for assigning your Accessibility/SEO rating, particularly for Drupal.

For starters, Drupal 7 SEO is pioneering the roll-out of RDFa functionality, which is the future of semantic web technologies (and therefore SEO). You make no mention of this.

Also, the community has some of the strongest commitments to accessibility targets, and it's had a central part in the conversation at many conferences and camps.

I understand the need to be critical (and I'm not opposed to criticism of Drupal), but your assigning it the lowest rating of the major CMS's (for Accessibility/SEO)... well, it gives the impression that you had no one taking part in your study who was invested in the Drupal community. This is the first I've heard of your organization, and it doesn't give the impression that you're particularly up to the task of what you do.

It honestly appears that you simply evaluated the out-of-the-box experience of Drupal, without even the most basic SEO modules installed. The community tries to minimize the core install profile to the essentials, and so your methodology is particularly unfair (intranet sites don't need SEO, after all).

Anyhow, I'm sorry if this seems harsh, as I imagine you worked very hard on it. (And your cause is noble, after all.) I just think your community deserves something more polished. And by "polished", I don't mean simply more favourable to Drupal. What I mean, is that I expect evidence of an in-depth analysis, with consultation of users who work with the platforms themselves. Even as a self-trained Drupal user (with no official computer science background), I simply can't believe that this was the case from reading portions of your report.

Thanks very much for hearing me out.

Kind regards,
Patrick Connolly

Laura's reply:

Hi Patrick,

I'm sorry you were disappointed in the report. In terms of having no one involved from the Drupal community, though, nothing could be further from the truth. A representative from Acquia (founded by Dries Buytaert) was our primary contact for Drupal for the report, and demoed the version of the system, complete with plug-ins, that they implement. We also wide review on our review criteria by a dozen or more people who specialize in Drupal (in fact, more than on any of the other four systems). If you'll take a look at the methodology and the contributors section, you can see exactly who was involved.

And our criteria for assessing the ratings is also clearly defined. While the Drupal community may be committed to accessibility targets, the current system is not currently particularly accessible. The core doesn't ship with any accessible themes, and it's quite challenging to bring the administrative interface into compliance with the 508 standards we rated against. In terms of SEO, semantic web support was not mentioned by any of the nonprofit consultants or staff members we interviewed to gather criteria for the SEO, and so is not included. The detailed criteria specifies precisely how Drupal compares against our criteria and our other criteria, including with add-in support, in this area -- please consult the details . The main concern about Drupal we have in the SEO area is that URLs are not human-readable without the use of a plug-in. As there's no particular value in having them NOT human readable, it's seems unnecessarily unfriendly to SEO to make people realize that they need human readable URLs and to find the plug in --- as we're reviewing for nonprofit organizations, many of whom are not very technically sophisticated, our review criteria includes thought as to how likely a non-expert is to actually get a solid SEO site out of the system.

As you can see from the report, we have corporate support from Phase2, Trellon, EchoDitto, and more -- all firms specializing in Drupal implementation -- I think with a more careful read of the report, you'll see that we in fact had a very detailed methodology which included substantial input from many folks with Drupal experience.

Laura

--
Laura Quinn | Executive Director | Idealware

Here are some notes towards a response...

1. Emphasis on features available in "core" creates a bias
   The emphasis on what is included or not included in the "core" of a system creates a bias towards systems that include more features/options/usability built into their default installation. This means that a system like Drupal actually gets dinged several times for the same basic problem. We get dinged for being complex because we require lots of configuration and additional modules, and we get dinged for being less accessible because we don't include those modules. And we don't get any bonus points anywhere for having a "small core"!! ;-)

2. Section 508 is inadequate to evaluate accessibility
   Everett's point about the inadequacy of Section 508 that Joe quoted at the start of this thread remains sound. When you review the list of consultants and authors in the report, it appears fairly obvious that this report is intended for the U.S. market: there are scores of U.S.-based consultants, but only 6 others: 4 Canadian, 2 international. That explains why they are using Section 508: because that is the current law in the U.S., regardless of how outdated it is. Outside of the U.S., the WCAG 2.0 is the current standard, though many countries (including Canada, which is mine) still refer to WCAG 1.0 in their published requirements. For the report to be useful to organizations outside of the U.S., it should refer to international guidelines.

3. Multiple languages not addressed
   I don't think there is any mention of sites running multiple languages in the report. In the U.S., non-profits must increasingly deal with audiences that speak different languages. For many non-profit/charity/government agencies in Canada, being able to produce a bilingual site is not a "nice-to-have" but a deal breaker in a CMS since Canada is a bilingual country. My impression is that Drupal's developments towards internationalization/multi-language capabilities are pretty good, and I'd be surprised to learn that the other open source CMSes can match Drupal in that area. This might be an additional criterion that would be valuable to encourage Idealware to add to future reports.

Phil.

I would add to my comments posted by Joe above, that the concept of an "accessible theme" would not be accepted by the majority of the accessibility field. The idea should be one of universal design, where users are required to make little to no choices or customizations in order to have an accessible experience. The evaluation would have better assessed the accessibility of the default themes (perhaps it did) rather than testing an "accessible" alternative included in the core package.

Drupal 7, I will admit, would lose points here for requiring some users to take special actions to disable the Overlay. However, I hope that this would be mitigated in part by the special attention that we have payed to ensuring that this action was as simple as it could be.

One further note on the ratings that I would add is that the rating terms may be misleading to some when compared to the criteria. I am basically restating what I wrote earlier about how the emphasis on which features are available in "core" or default installs creates a bias. But now that I've thought a bit further about it, I think it is not just a bias, but also that the terms used are misleading based on what is being evaluated.

In the area of Accessibility/SEO, I would argue that the criteria stated in the Idealware report are not actually sufficient to evaluate whether one CMS is "excellent" compared to another which is "fair". If I understand the criteria correctly, the ratings are not actually evaluating whether one CMS is more accessible than another or if one has better SEO than another, which is what one would expect by the use of terms "fair" vs. "excellent" for example. It isn't like the terms compare to something like WCAG "A" vs. "AA" vs. "AAA" or Prioirity 1,2,3, which is also something one might expect in an evaluation of accessibility.

The ratings are rather set up as binary evaluations: is the CMS accessible? does the CMS have SEO features? If your CMS is judged as 508 compliant, then that's it, you meet 508 and you are accessible. Does your CMS provide human-readable URLs, then that's it, you are SEO-friendly. The report does not really judge whether one CMS is more accessible than another or better for SEO than another. The question of whether or not one receives a "fair" vs. "solid" vs. "excellent" rating is largely concerned with whether the features in question are part of the core set of features or whether additional modules are required.

So for example, there is a criterion related to whether the administrative interface is accessible or not. As you move from "fair" to "excellent" you do not move from an administrative interface that is least accessible to one that is most accessible, you rather move from one that requires custom work, to one that can use a community theme, to one that has a theme that ships as part of its core.

Likewise for SEO, the criterion that Laura mentions as causing Drupal's rating to be only "fair" is because Idealware thinks that clean URLs + pathauto should be installed by default. It is not an evaluation of whether Drupal is "fair" with respect to SEO compared to Joomla being "solid", but rather an evaluation of whether one particular SEO feature is part of a default install.

Viewed from this perspective, then, the terms are misleading. The ratings do not accurately reflect a scale of relative accessibility or SEO-friendliness, they just reflect how mcuh configuration you may have to do. Drupal 7 could very well be the most accessible of all the CMSes evaluated, but it would not be reflected by this evaluation scheme since the degree of accessibility is not what is being evaluated.

Phil.

Just an update here as they are working on the 2012 report. I've added a comment about accessibility as they are still sticking with the old Section 508 compliance.

As you may have heard, Idealware is currently updating and expanding our 2010 Content Management Systems Report. We'd love your help in thinking through the criteria we will use to evaluate these systems, especially in comparing proprietary as well as open source systems.

Review and comment now on the CMS Criteria Review wiki.

For 2012, in addition to updates of the four open source system reviews (Wordpress, Drupal, Joomla and Plone), we’re adding detailed reviews of seven new systems, both open source and proprietary (dotNetNuke, eZPublish, ExpressionEngine, SquareSpace, Ektron, Convio, and Blackbaud’s NetCommunity).

We'd love your comments and thoughts on the review criteria, to ensure we're reviewing for the things that matter to the nonprofit community and those that will help us to usefully compare the pros and cons of the systems. The wiki will be open for comments at least this week and next -- so comment early and often!

Review and comment now on the CMS Criteria Review wiki.

Have questions or concerns about the criteria review process? Contact kyle@idealware.org

Thank you for your help!

Laura
--
Laura Quinn | Executive Director | Idealware
laura@idealware.org | www.idealware.org

I do think it is useful to find some good criteria for objectively comparing CMS solutions. Not that we're an objective body, but...