If there's one thing we know that's beneficial for making good products it's usability evaluation, right? So - being the usability zealot that I am - I felt an immediate urge of disagreement hitting me when I saw the title of the CHI2008 paper Usability Evaluation Considered Harmful (Some of the Time)* (pdf) by Saul Greenberg and Bill Buxton. But fortunately I did read the article and discovered Greenberg and Buxton lay out a very compelling argument against usability evaluation being misused as the standard method to evaluate any design, including innovative, conceptually new user interfaces.

Formative versus summative testing
The gist of it is that there's no use in comparing the usability of an immature tangible interaction concept (say, the Wobble Lamp) against an element of a mature on-screen user interface (e.g., Windows 98). It's the innovator's dilemma: you might dismiss a new technology or concept because it doesn't perform sufficiently, but it only underperforms because it is immature; the potential of the immature might be much bigger than that of the existing (see illustration above). You want to know and show what's good about your design, its potential, and what can be improved to reach it. And that is not something that only performing a usability evaluation will give you an answer to.

Abstract
Current practice in Human Computer Interaction as encouraged by educational institutes, academic review processes, and institutions with usability groups advocate usability evaluation as a critical part of every design process. This is for good reason: usability evaluation has a significant role to play when conditions warrant it. Yet evaluation can be ineffective and even harmful if naively done ‘by rule’ rather than ‘by thought’. If done during early stage design, it can mute creative ideas that do not conform to current interface norms. If done to test radical innovations, the many interface issues that would likely arise from an immature technology can quash what could have been an inspired vision. If done to validate an academic prototype, it may incorrectly suggest a design’s scientific worthiness rather than offer a meaningful critique of how it would be adopted and used in everyday practice. If done without regard to how cultures adopt technology over time, then today’s reluctant reactions by users will forestall tomorrow’s eager acceptance. The choice of evaluation methodology – if any – must arise from and be appropriate for the actual problem or research question under consideration.

Alternative methods
As alternatives to usability evaluation the authors suggest design rationale, scenarios of use, case studies, and participatory critiques. I would add user experience sampling, observational research, and in-depth interviews to that. Greenberg and Buxton do a very good job at pointing out that the evaluation of innovative interaction concepts should be formative, not summative, and that usability is just one of the product properties to evaluate a product by.

Read the paper: it's a mind-changer.

See also
> Greenberg's slideshare set of the CHI presentation of the article

Reference
*Greenberg, S. and Buxton, B. 2008. Usability evaluation considered harmful (some of the time). In Proceeding of the Twenty-Sixth Annual SIGCHI Conference on Human Factors in Computing Systems (Florence, Italy, April 05 - 10, 2008). CHI '08. ACM, New York, NY, 111-120.

[Illustration: MIT Aurora project]