Basing Design Decisions on “Those Six Guys”

As I explained in yesterday’s post, one of the core features of RDFa is under attack in the HTML Working Group. I use the phrase “under attack” loosely because I can’t imagine that the Chairs for the HTML Working Group are going to remove CURIEs based on Ian Hickson’s (the editor of the HTML5 specification) current proposal to eviscerate RDFa. But hey, stranger things have happened. I’m concerned about this possibility because I’m also the current Chair of the RDFa Working Group. Removing CURIEs would break all of the currently deployed RDFa content out there (430+ million web pages), so I doubt it’ll be removed. Personally, I think that the case against CURIEs is so weak that it is laughable… or cry-able. Honestly, I switch how it affects me from day to day – working on standards does that to you, it makes you manic. Today, it’s laughter – so let’s laugh a bit, while we can.

Ground Rules

Fair warning: This post is probably going to get a bit ranty from this point on. I try not to rant publicly too often because it can be construed as petty whining, or at worst, used as a reason to divide communities and people. So, know that this rant is mostly petty whining with a sprinkling of irony. It’s hard to not understand what’s going on here and not have a good laugh about it. This rant is mostly about a few key individuals in the Web standards community and how, at times, fantastically ridiculous things can emerge from multi-year conflicts between them. Nerd fight!

I also don’t want this to come across as a slam against Ian Hickson. I respect the guy because he’s moving HTML5 forward and he’s pissed off enough people along the way to make people passionate about Web standards again. I like that people get excited about HTML5, even though many of them don’t know what in the hell it means. Hell, even I don’t know what it means other than it’s supposed to cure this nasty rash I picked up in Nicaragua.

Back to our favorite non-benevolent, benevolent dictator. I don’t appreciate many of the political tactics and doublespeak Ian uses in the name of moving the Web forward, but I tend to not care about most of the crap he pulls unless it causes the good people in the Standards Community grief. There’s even a really fun (NOT SAFE FOR WORK) website that follows the many antics of Ian Hickson and friends. So, hat’s off to Ian for getting shit done.

I Do Science. Now You Can Too!

Now, let’s have some fun. One of the arguments that Ian has been making ever since we pushed for RDFa to go into HTML5 goes something like this:

“In a usability study for microdata, it was discovered that authors in fact have no difficulty dealing with straight URLs rather than shortening them with prefixes.”

The usability study alluded to was done by Google and was used to determine a few of the features that one can find in Microdata (which is a competing specification to RDFa in HTML5). When it was performed, Ian was quick to point out that a study had been performed by Google on Web developers not having an issue with typing out full URLs, but no data was released for many months. Some of the reasons cited were privacy concerns, it was an internal Google study, “I know better than you”, etc.

However, that didn’t stop Ian from referencing the usability study when discussing what he deemed to be faults in RDFa. In fact, the blog post referenced by the change proposal (which was put together fourteen days ago) to remove CURIEs from RDFa was the first time that I had ever seen the raw data, or the number of people that had taken part in the study. I passed it by others in the community and it was the first time that they had seen the data as well. There are many, many people that track this stuff and the fact that this was posted in October 2009 and we are just now seeing the data is… well, I just can’t explain how the entire RDFa community missed this vital piece of information that we have been looking for for the past two years. Anyway, there it is – all the data on the thorough set of tests run by Google, across potentially hundreds of participants, all for figuring out how key features of Microdata would work.

Bah – Confidence, Schmonfidence.

One of the first things you look for when somebody alludes to a “scientific study” are the number of participants. The other thing is “confidence level”, which is how sure you can be that what you find is in common with the general population you’re testing. So, a quick calculation would show us that if we had 100,000 people in the world that write HTML by hand on occasion, and if we wanted to be at least 90% confident of what we find, that we would need roughly 383 people to participate in the study.

So, I scan the study and find them… all six of them, who will henceforth be known as those six guys. Note: I’m using the gender-neutral form of guys so as to not be a sexist asshole.

Those six guys are far less than the 383 people that you would need for a decent scientific study. Design decisions were made for Microdata based on those six guys. There were supposed to be seven people partaking in the test, but one of them was a no-show. Perhaps it was because they knew some basic statistics and understood that the study was a waste of their time, perhaps it was because they didn’t want to feel the pressure of making a design decision for the billions of people in the world that use the Web, who knows!

Sure that I had missed something, I read through the study again. I re-checked my calculations on sample size, attempted to figure out what the confidence level for those six guys could be when applied to smaller populations, everything was pointing to something fantastically wrong having happened. Here we were, the RDFa Community, having to defend ourselves against an unknown Google study over the past several years where we believed everything had been done with the mythical exacting precision honed on every problem to cross Google’s path. There was some sort of deep Google A/B testing that was applied to this study, of this, we were certain.

Take a wild guess at how confident you can be with a population of 100,000 people and then sampling only 6 of them? By sampling those six guys, your minimum confidence level is 54%. That’s almost equivalent to the confidence you get by flipping a fucking coin. Those six guys don’t represent science, they represent random chance. You could put together a test with three donkeys and five chinchillas and be more confident about your findings than the Microdata Usability Study. I suggested the donkey-chinchilla metric to the engineering team at our company and like most of my brilliant ideas, they chose to ignore it, or me. It’s difficult to tell when people refuse to make eye contact with you.

One important key to success is self-confidence. Another is not being wrong.

Ian goes on to draw conclusions from those six guys:

“One thing we weren’t trying to test but which I was happy to see is that people really don’t have any problems dealing with URLs as property names. In fact, they didn’t even complain about URLs being long, which reassured me that Microdata’s lack of URL shortening mechanisms is probably not an issue.

Wait a sec. “people“? “they“? Who in the hell are we talking about here? Are we talking about those six guys? I’m pretty sure that’s who we’re talking about, not the general Web Developer population. I showed this to some other folks that have a grasp of college-level statistics and it was fun to see them wince and then watch as their heads, figuratively, imploded. One absolutely should not draw any conclusions from the horribly flawed Microdata Usability Study. Ian had asked me a few weeks ago why we hadn’t done a usability study like the Microdata one and not having known the number of participants in the usability study, I replied that we just didn’t have the resources that Google did to carry out such a comprehensive study.

But here we are – a brave new world of “scientific inquiry”! So, I headed off with a spring in my step, determined to do as good of a job as the Google Microdata Study. I e-mailed, IMed, phoned and talked my way through until I had responses from 12 people, twice as many as the Google study! I asked them whether or not they found URIs hard to type and found CURIEs useful. BIAS ALERT: These are all people that have deployed or are successfully deploying RDFa as a part of a product. The answer in all cases was unequivocal: “Don’t you have anything better to do with your time? Why in the hell are you wasting my time with these questions? Of course CURIEs are necessary to ease authoring, isn’t it obvious?”

So, take that Microdata Usability study’s 54% confidence level! Based on my in-depth study, I’m at least 65% confident that CURIEs are useful.

Yes, I’ve done Hallway testing and yes, I’m aware of things like the Nielsen/Landauer formulas for Usability Testing. Yes, I think that Usability Testing is very important – when done correctly and with a complete set of alternatives. A single test with six people does not qualify. So, what’s the lesson here: Oh yes – it’s that anecdotal evidence and studies based on those six guys are worth as much as the time it takes to slap them together. That is, it’s next to worthless.

What is worth something is hard data that is statistically significant – such as there are at least 430+ million web pages containing RDFa and CURIEs today. That there are currently 23,913 RDFa-enabled Drupal 7 sites using CURIEs right now, which will grow to 350,000+ sites in 2 years. That Google, Yahoo, Bing, Facebook, Flickr, Overstock.com, Best Buy, Tesco, Newsweek, O’Reilly, The Public Library of Science, the US Whitehouse, and the UK Government among tens of thousands of other websites are successfully using RDFa and CURIEs.

Remove CURIE support from HTML5+RDFa and most of those site’s meta-data will go dark. I’m laughing about the prospect of that today, but only because it seems laughable. If CURIEs are removed from HTML+RDFa, I cannot imagine the shit-storm that’s going to rain down on the RDFa Working Group and the HTML Working Group. Haha. *sob*

8 Comments

Got something to say? Feel free, I want to hear from you! Leave a Comment

  1. Julian Reschke says:

    From the W3C HTML5 spec (*):

    “Thanks to the participants of the microdata usability study for allowing us to use their mistakes as a guide for designing the microdata feature.”

    Maybe that should be changed to say “to those six guys”. :-)

    Julian

    (*) Yes, the HTML5 spec, not the Microdata spec.

  2. Shelley says:

    One of the more enjoyable rants I’ve read. You should rant more often.

  3. Shelley says:

    I also want to mention that yes, I had seen the study, as my comment in the blog post demonstrates. And yes, I have brought up these concerns in the last, oh, two years.

    Such as

    The study precursor

    http://lists.w3.org/Archives/Public/public-html/2009Jun/0801.html

    http://www.mail-archive.com/www-archive@w3.org/msg02703.html

    Oh, yes, I was told I didn’t play well with the other children on this one. In fact it was about six of those guys who all took me to task for my bad attitude.

    Notice Ian’s talk about bias with prefix–biased before the test, biased at the test.

    And here

    http://www.brucelawson.co.uk/2009/this-millenium-in-html-5-politics/

    http://lists.whatwg.org/htdig.cgi/whatwg-whatwg.org/2009-August/022215.html

    Aftermath

    http://lists.w3.org/Archives/Public/public-html-comments/2010Dec/0007.html

    Anyway, I could go on, but won’t.

  4. Any reaction from Ian ?

  5. So Hixie found 6 Hello World capable ape-men. :P As self-helping computer professional composing provenance advanced than Hello World, prefixes redefinable from localhost to production domain as opposed to PURLs are fundamental need where content composition compels presence of mind.

  6. If WHATWG deals with “living” standard and CURIEs are living, then which standard does its HTML5 spec editor work on? (Hint: Complex= Real+Imaginary “j”)

Trackbacks for this post

  1. The Familiarity Trap and Why The WhatWG Are The Wrong Cowpath To Follow - Thomas Legg's Geek Blog

Leave a Comment

Let us know your thoughts on this post but remember to play nicely folks!