June 17th, 2008
By Christine Perfetti - July 29, 2002
Derek has worked on community features for Netscape, Nike, and Sony, along with creating the community sites, {fray}, Kvetch!, and SF Stories. Christine Perfetti, a consultant at User Interface Engineering, recently talked with Derek about his experience. Here is what he had to say about creating effective online communities:
UIE: We hear the phrase, “Virtual Community” thrown around all the time. How do you define “Community”?
Derek: I advise clients to never call their sites “communities.” Instead, I tell them to provide adequate tools for your members to communicate with each other, plenty of relevant material to talk about, and an elegant structure that encourages conversation. If you’re successful, your members will start calling it a community on their own.
But since community is a personal business, I’ll give you my personal definition of the word: “Web communities happen when users are given tools to use their voice in a public and immediate way, forming intimate relationships over time.” I think that covers all the bases. Really what it’s about is power: As the site owner, I’m giving away some of my power to my audience, to give them a voice on the site. And that’s really a leap of faith sometimes. But when it works, the benefits can be astounding.
What are the major benefits for site owners considering giving some of their power away? In other words, why should designers even be thinking about community features?
There are huge benefits. Studies have shown that community site participants are far more likely to spend money online. And the connection a community member makes with a brand in a community setting can be intense. There’s no better way to form a relationship with your audience, and to enable them to form relationships with each other.
What common misperceptions do people have about designing community spaces?
The biggest misconception is that community can be built at all. It can’t. What you can do is build an environment that is conducive to social interaction. If people adopt it and make it their home, they’ll call it a community for you. Community isn’t built—it’s grown. That’s why the specifics of that environment are so important. The visual design, interaction design, experience design, text, signage … all of it has to work together to make the user feel comfortable and give them clear and welcome ways to communicate with other users.
In 1996, you started a storytelling web site called fray.com. You’ve also created two other sites with community features: Kvetch! and San Francisco Stories. What was your biggest take-away from these experiences?
I’ve learned that there’s a huge hunger out there for true stories from real people. And there’s no better place to collect those stories than the net, where everyone can have a voice. But I’ve also learned the hard way that crafting environments that produce positive, creative contributions and discourage negative, destructive ones is a tricky business. It involves creative writing, visual design, experience design, and a whole lot of social psychology.
Probably the biggest question we get from clients about Design for Community is how they can get community features to work for them. What advice would you give these people?
The first step is to examine your own motivations. Why do you want a community area? What are you hoping to gain? What are you offering to your audience to entice them to participate? What will they get out of it?
It’s difficult for clients, but the first step is to visualize what your community area is going to look like a year from now. Once you’ve got the vision, it’s a whole lot easier to get from here to there.
The thing to remember is that there is no magic formula. Every site is different, every audience is different, and getting that audience to interact is different. If it were just a matter of plugging in a piece of software, well, my book would have been about 290 pages shorter.
In your book, you discuss major successes of companies striving to create online communities. Who are some of the biggest success stories? How did these sites encourage community?
The clear winners are places like Amazon and eBay. Now, these aren’t the community ideals. Let’s face it, they’re both about buying and selling at their core. But where other web stores failed, they succeed. In both cases, it’s because they designed their sites around the strengths of the Internet: social connections.
eBay and Amazon both act as social connectors with simple, but effective, reputation management and discussion tools built in. If I stiff someone on eBay, I’m going to get punished. Not by eBay, but by my fellow members who will rate me down. And Amazon learns a lot about me—who my friends are, what I like, what they like—and then caters my experience toward that.
Speaking of Amazon, we often hear people say that one of Amazon’s best features is the customer reviews. But many customers just read the reviews and go buy elsewhere. Can Amazon view this community feature as a success if it doesn’t result in people buying?
I’d say that if people are thinking of Amazon as “the place to go when I’m thinking about books” then they’re a success. Because, eventually, that person is going to hit the buy button—it’s just too convenient.
Amazon’s real power is in the Friends and Favorites program. Just tell them who some of your friends are, and then you get this custom page that’s like the local newspaper for a small town where you’re the mayor. “Derek! You have two friends with birthdays coming up! Want to see their wish lists? And hey! Four of your friends bought the new Radiohead album! Want to check it out? And hey! Your buddy Lance just posted a review of this espresso machine. Want to read it?”
Do I care that 100 people I don’t know gave a book a good review? Yeah, a little. Do I care that one of my friends liked the book? Oh, yeah. A lot.
Successful Internet companies are going to be the ones that act as this kind of virtual middleman. It’s one of the things the Internet does very well, and no other media can replicate.
Are there any major mistakes site owners should avoid?
The number one mistake I see is when organizations simply forget to ask themselves why someone would ever want to participate in their community features. It’s an important question. As a user, it takes time and effort to create a membership, tweak preferences, and interact in some way. The user has to feel like they’re getting something back. If they don’t, why bother?
Sometimes when you’re on the inside looking out, it seems obvious to you why someone would want to participate. But that needs to get communicated to the user or it won’t work. The number one piece of advice, for designers and managers alike, is: “think like a user.”
We have clients that tell us they’re designing a web site for “everyone”. In your book, you argue that all communities, to a certain extent, should exclude some of the users. You say that providing a “barrier to entry” can help strengthen a community. Putting obstacles in the way of some users may seem counterintuitive to many designers. Why would explicitly excluding some users strengthen a community?
Imagine going to a party. You’re standing in the living room, holding a drink, talking to a few people. Then imagine that a football game lets out down the street. Hundreds of people are now streaming towards your party. Would you want the front door open or closed?
The simple truth is: you can’t have a good conversation with 300 people. You can’t really have one with 30. Good conversations happen in small groups. So on the web, your task is to set a barrier to entry that is high enough to have good conversations, but low enough that the people who are really motivated find it easy to get in.
And there are many different ways to set a barrier to entry. Some are formal (mandatory membership creation, for example), but you can also use informal barriers. Sometimes simply moving the link to the discussion area to the bottom of the page is barrier enough. That says to the user: If you’re going to come in here, you really should have read the above text first.
The thing to remember is, the Internet is a big place, and there is indeed a community out there for everyone. But your community doesn’t have to include everyone. The idea is to find the people who care about your product, service, or niche, and make a place just for them.
Thanks Derek.
(Note: This article was originally published on User Interface Engineering’s web site.)
Posted in Articles - No Comments »
June 17th, 2008
By Christine Perfetti - December 14, 2005
When prospective college students arrive on the University of Northern Iowa’s web site, they can read all about Jen. Jen is a current student at UNI who shares some of the reasons she chose to attend the university.
Jen shares her experiences as an UNI student.
UNI.edu’s designers have put considerable time and effort into sharing the experiences of people associated with the university. The site has more than 40 detailed profiles of students, faculty, staff, and alumni.
There’s a huge amount of information users can glean from these profiles. For example, I bet you didn’t know Entertainment Tonight’s co-host, Mark Steines, attended the University of Northern Iowa.
A profile of Entertainment Tonight’s Mark Steines
Why did UNI’s site’s designers invest so much energy in these profiles? One reason is they know users need this type of content to make a decision about where to attend college. The designers’ research told them that, before choosing UNI, prospective students visited the web site with one main goal — to see what other students had experienced. The site’s designers do a great job offering users what they needed most: Inukshuk content.
Inukshuk Content Offers Reassurance
At User Interface Engineering, we use the term “inukshuk” to describe the type of content found in the UNI profiles. It’s the name of stone figures created by Inuit hunters as guide markers. The hunters arranged the piles of stones in the likeness of human beings, signifying to other hunters that someone already has passed through and experienced their same journey. An inukshuk provided reassurance and empathy to others and alleviated an Inuit’s fear that they were off the track.
In our research, we’ve seen that, just like the Inuit hunters, users on the web want reassurance that others have shared their experiences. Many times, users are contemplating important decisions. To understand if they are making the right decision, they often want to go beyond facts. They want to know if it “feels” right.
Inukshuk content, when done well, can give the user confidence in a way that factual content can’t. The Inukshuk content assures the users that other people have been there before, following the same path they are currently traveling. The users see that these other folks have made it through the decision process and can gain reassurance in their own decision.
Amazon.com Leverages Inukshuk Content
Amazon.com has massive amounts of information devoted to helping users make informed purchase decisions. Here’s a typical example of a technical review found on Amazon:
In this technical review on Amazon.com, the customer describes the features of their video camera.
The technical review’s purpose is to give customers a venue to share the pros and cons of a product they’ve purchased. It’s a very fact-based discussion of the product. In this instance, the customer describes the features of the digital camera he purchased. This is typical of the technical reviews we see on Amazon, where the customers relay product facts.
Aside from the technical reviews on Amazon, there are other customer reviews where the product’s features aren’t discussed at all. Instead, the writers focus on telling other readers about their particular experience with a product and how it worked (or didn’t work) for them.
For example, a digital camera purchaser talks about how the camera enabled her to share pictures of her new baby and how it exceeded all of her expectations. We call this an Inukshuk review. Customers focus on their specific experiences with the product, with almost no technical information.
Amazon’s inukshuk reviews focus on the user’s experience with a product.
Inukshuk reviews provide users with important information they wouldn’t get from reading a technical description — the experiences of other users who faced similar decisions. We’ve seen that Inukshuk content is just as important to users as the technical reviews. Both technical and Inukshuk reviews are vital in helping users make purchase decisions.
Inukshuk Content Found on All Types of Sites
We see successful Inukshuk content on all types of web sites. Procrit.com is a site designed to provide users with information about Procrit, a drug treatment for people suffering from anemia.
When users arrive on the site, they have many questions about the drug treatment: What are the side effects for people who are using the drug? How happy have patients been with the regimen? How tired will I feel while taking Procrit??
To address these concerns, Procrit.com’s designers offer users videos with actual patients who’ve been treated with the drug. The purpose of these videos is to expose potential patients to others who have been in the same situation and reassure them that they will be okay. In our testing, users were much more confident about the drug after hearing these real-life stories.
Procrit.com’s patient videos
Fidelity.com’s design team used a similar tactic when trying to persuade job seekers that Fidelity is the right place to work. The site has an entire section devoted to current employee testimonials, offering users exactly the type of information they need at the time – reassurance that Fidelity is a great employer to consider.
Fidelity.com’s job opportunities
Finding Opportunities to Reassure Users
We’ve found that the most successful sites create Inukshuk content to help users feel confident in their important decisions. The content can take many different forms, including videos, testimonials, user diaries, reviews, ratings, and discussion boards. Regardless of the form, the key ingredient of Inukshuk content is that it lets your users know that they aren’t the first to deal with their specific types of issues. Others have traversed this way before.
(Note: This article was originally published on User Interface Engineering’s web site.)
Posted in Articles - No Comments »
June 17th, 2008
By Christine Perfetti and Lori Landesman - Jan 31, 2001
We hear all the time from web designers that they spend countless hours and resources trying to speed up their web pages’ download time because they believe that people are turned off by slow-loading pages. Their concerns have been amplified by experts like Jakob Nielsen who asserts that users become frustrated after waiting too long for pages to load. It makes sense that a slow loading page is unusable. We know that if a page takes 2 hours to load, chances are people will abandon their tasks. But when does download time go from too slow to fast enough?
Nielsen reports that the home pages of the most popular sites he studied took an average of 8 seconds to download, whereas the pages of the less popular sites took an average of 19 seconds to download. He therefore concludes that users will be annoyed or frustrated by pages that take any longer than about 10 seconds to load.
When we began our research, we thought we would find a strong relationship between page download time and usability: sites with faster download times would be more usable than slower sites. We also expected that users would be consistent in their ratings of site speed, and that these ratings would correlate strongly with the actual speed of the sites.
To test these predictions, we studied 10 different web sites over a 56 kbps modem. On these sites, we had users perform their own personal tasks; each user did something that was interesting and meaningful to her. No two users performed the same tasks on any site. For each of the sites, we had users rate how fast they felt the site was. We called the users’ measures their “perceived speed” of the site. Later, we watched videotapes of the studies and measured the actual download times of the pages.
We started by confirming one of our hypotheses: all users rated the speed of the 10 web sites consistently; they thought Amazon.com, REI.com, and L.L. Bean.com were the fastest and About.com was the slowest. Despite having performed different tasks on these sites, users were consistent in their reports of perceived speed.
Our other finding, though, took us entirely by surprise. When we looked at the actual download speeds of the sites we tested, we found that there was no correlation between these and the perceived speeds reported by our users. About.com, rated slowest by our users, was actually the fastest site (average: 8 seconds). Amazon.com, rated as one of the fastest sites by users, was really the slowest (average: 36 seconds).
There was still another surprising finding from our study: a strong correlation between perceived download time and whether users successfully completed their tasks on a site. There was, however, no correlation between actual download time and task success, causing us to discard our original hypothesis. It seems that, when people accomplish what they set out to do on a site, they perceive that site to be fast.
When we thought about these findings, they made a lot of sense to us. If people can’t find what they want on a site, they will regard the site as a waste of time (and slow). But, when users successfully complete tasks on a site, they will perceive their time there as having been well spent.
Jakob Nielsen tells designers to focus efforts on improving actual page download times on their sites. But what we’re seeing leads us to wonder if it’s worth the resources to make web pages load like lightning. Instead, we’re wondering: When users are complaining about the download speed of your site, what are they actually complaining about? Are you better off making the site load faster or ensuring that users complete their tasks?
(Note: This article was originally published on User Interface Engineering’s web site.)
Posted in Articles - No Comments »
June 17th, 2008
By Christine Perfetti - July 26, 2007
In his research, Scott Berkun, the author of the popular new book, “The Myths of Innovation,” has done a fantastic job of demystifying innovation and debunking dangerous assumptions about how breakthroughs happen. UIE’s Christine Perfetti recently had the chance to talk with Scott about his new book and his research in the area of innovation.
UIE: In your book, you discuss the misconceptions about many of the biggest innovations in history. For example, you mention that Newton didn’t discover gravity by watching apples and Thomas Edison didn’t invent the light bulb. Why do you think these false beliefs are still so popular and memorable?
Scott Berkun: One of the biggest reasons is that the myths are fun. We all love stories that entertain or mystify us, and it’s natural that given the choice between a dull story for how something happened, and an odd, curious or funny one, we’ll tend to want to hear, and tell, the latter.
There’s nothing wrong with this, unless you actually want to learn how to innovate: in which case we have to dig deeper and find out the truth. That was the primary goal of my book.
When you use the word “Innovation,” what do you mean?
It’s funny, I spent a great deal of time looking at different definitions of the word, but in the end decided not to bother with a long-winded exploration of what the word means.
In my book, I’m deliberately sloppy with the definition. Invention, discovery, innovation, creative thinking, and progress are all in the same ballpark and that’s the field of myths the book explores.
In your research on Innovation, you debunked the myth that the best ideas win if the design is better than its competitors. Can you give an example of the best idea failing to win?
I think it’s pretty rare that “the best” idea among experts in any field becomes the dominant, mass popular leader. HTML is not the “best” programming language. Certainly few computer scientists believe Microsoft Windows is the best operating system, and very few doctors believe Airborne is the best cold remedy. In my research, I’ve explored all the factors that contribute to innovation adoption, and surprisingly only a few of them have to do with the abstract quality of the idea behind the innovation itself.
What advice would you give managers who want to encourage creative environments and innovation within their team?
There’s no magic secret. Most of the teams I researched that did well had smart and motivated people and an environment that rewards those people for taking risks, experimenting, and using their own initiative. It’s also important for the team members to have clear goals or problems that needed to be solved. If a manager can truly provide these things, they’re ahead of most of the working world.
Are there any circumstances where you would recommend teams avoid focusing on designing for innovation?
Yes! Thanks for asking! I think innovation is overrated. Customers don’t care about how innovative you are. They just want to be happy and satisfied. And that’s about good design.
The best advice I can give is to focus on people and their problems. Few great innovators worried about anything else. The fact that they found a new idea had more to do with their passion for solving someone’s problem than anything else. Innovation is a huge distraction these days. That’s one of the myths I hope people will understand how to dispel from reading the book or attending my seminars.
In your opinion, what are some of the obstacles that prevent design teams from creating innovative products?
The biggest obstacles are political and psychological. Design teams are sometimes so detached from what the business or engineering leaders are doing that their big ideas are stillborn - the designers aren’t innovating in ways that the rest of their organization could possibly be receptive to.
Some call this politics, but that’s half of the innovation game: lining up the resources and relationships needed to support a brilliant design idea. There are countless innovations that never became products because their champions ignored the politics necessary to bring their ideas to the world.
On the psychology side, designers can be surprisingly conservative and almost shy about presenting their ideas to others. They’re so afraid of criticism that they never allow their passions to show, especially in a confrontational situation, which ironically is exactly where their passions might be of the greatest use to them. One chapter in my book focuses on the psychology of ideas, and how creators always face the same criticisms and challenges. Being aware of those challenges and preparing oneself to handle them gracefully is a large part of being a successful innovator.
What are some of the reasons innovation efforts fail in an organization? Are there some common mistakes you’ve encountered?
The two big ones are 1) a failure to let ideas grow and 2) an unwillingness to take risks. In the first case, ideas are like plants - the seeds don’t look much like the final flower, and need time and nurturing to blossom. If there’s no incubator in an organization, there’s no way for new seeds to develop, and therefore, not much innovation is going to happen.
In the second, innovation is change by definition. No matter how you define it, it means doing something different. This demands risk. The bigger the innovation, the bigger the risks. Any organization that claims they want innovation but isn’t willing to take risks is lying to itself.
For innovation to thrive within an organization, you’ve argued that it’s important for someone in power to support the project. How would you recommend designers go about getting buy-in from that person?
Yes, if the person guarding the door between you and your customer doesn’t at some point support the project, there’s really no point, is there? If they’re going to say “no,” then all your great ideas won’t help anyone.
There are three easy tricks for getting buy-in.
1) Watch others who got buy in. How did they do it? Why did they get support? What questions did they ask? Every executive is different and the best advice is to watch your VP, see who they grant buy-in to and why, and learn from their successes and mistakes.
2) Go and ask for it. There’s no reason to be shy. If you have an idea in progress that you want support for, say, “Hi Mr. VP. I have a proposal for X that needs your support. What’s the best way to get your feedback on this?” Often a 30 second 1-on-1 conversations yields more useful information than a 10 minute presentation in a big meeting
3) If #2 is scary, go to whoever you saw do #1 above and ask them for *their* input.
In your book, you talk about Flickr’s success. Yet, Flickr was originally conceived as an online game, The Game Neverending. What can design teams take away from Flickr’s story?
The lesson is simple: never stop asking, “what else might this be good for?” Design is the application of ideas to problems, and even if you’re half done designing a toaster oven it doesn’t mean that the same design might not be useful for something else as well (say, a doorstop, a boat anchor, or a footwarmer for Eskimos).
At a higher fidelity, it might even be a great sales tool or in some cases, a way to explain to users what a tool does as part of the marketing. But the tool doesn’t replace wireframes, specifications, use cases and all sorts of other tools that we employ throughout the product cycle.
There’s a common belief that it’s more difficult to innovate in larger organizations. Have you found this to be true?
It is often harder but not because of size. It’s because of the diminished ability for people in large organizations to take risks. That’s the real problem. Many big organizations continually innovate: 3M, GE, even Google now can be considered a large organization. So, there are cultures and management styles that allow for innovation at any scale - and that’s the real thing to focus on - culture and management, not size.
Thanks, Scott!
(Note: This article was originally published on User Interface Engineering’s web site.)
Posted in Articles - No Comments »
June 17th, 2008
By Christine Perfetti - September 1, 2001
In today’s competitive marketplace, designers are faced with the responsibility of creating new and innovative products that will recreate or capture their market. But, how do they go about proactively developing a creative and innovative design?
Our clients are often surprised by our answer to this question. They commonly believe that designers derive new and innovative designs solely from their own creative inspiration. But our research shows that the most innovative designs are those that are created by designers when driven by customer data.
Customer Data Drives Innovation
Every so often, a designer does come up with a truly inspired, creative idea completely on their own. But designers often believe that this is the norm, not the exception.
Many designers understand that the need for customer input is invaluable, but they perceive the process to be too costly. Without the necessary customer involvement, designers end up developing products based on what designers believe customers need, rather than what they actually need. The resulting design becomes a solution looking for a problem.
Karen Holtzblatt, Co-Founder of InContext Enterprises, agrees that customer data drives innovation and that the most innovative solutions come from involving the customers. She believes that the most reliable road to creativity is when an innovator gathers customer data and observes problems that need fixing.
Karen says there are many real-life examples: Thomas Edison invented the light bulb when he’d observed people struggling by candlelight. WordPerfect was developed when the creators asked for feedback from their primary customers, office secretaries. She points out that products are only innovative if they’re solving a real problem for customers.
Success Stories: Customer-Focused Design
eBay’s web site is a prime example of innovation at work. Pierre Omidyar, eBay’s founder, came up with the idea of an online auction site by recognizing a specific problem his wife (then girlfriend) was facing — she needed a venue to collect and trade Pez dispensers. In the course of understanding his wife’s predicament, Pierre came up with the idea of an online auction house.
Look at eBay today. Along with selling Pez dispensers, the site sells any item a person could possibly want to buy. Pierre didn’t come up with this highly profitable idea “out of thin air”. Instead, he recognized a problem, had a vision, and most importantly, gathered constant feedback from customers throughout the design process. eBay’s innovation didn’t stop at the launch of the site. They constantly collect feedback from customers, helping them add dozens of new and innovative services and features every year.
At User Interface Engineering, we see many examples of customer data leading to innovation. For example, SAP asked us to visit some of their customers to observe how they used the software. When we visited these customers on-site, we observed that all of the companies had the same problem. After the SAP system was installed, SAP’s customers needed to train their employees on the product. Every SAP customer we visited asked the same question: are we really the first company that has needed training on this application?
After they identified the problem, SAP identified their customer’s training needs and created several innovative training solutions for their customers. Because they based the training on the work already done by their customers, this innovation was very easy to implement and cost almost nothing. But SAP would have never known there was a problem without talking to their customers.
Contextual Inquiry: A Technique for Practicing Customer-Driven Design
The exciting thing is that any design team can create innovative products if they learn some simple techniques. Designers with limited resources can easily implement most of these techniques on a small budget.
One technique we like to use to create innovative designs is Contextual Inquiry. Contextual Inquiry immerses product designers in actual customer data by having designers observe the work of users in their natural environment. Design teams can quickly identify specific problems and needs of their customers, One advantage of this technique is that it provides a framework for designers to synthesize the customer data they collect and use it to produce creative products.
No matter what techniques teams use to collect data, the important thing is that they believe that innovation can be accomplished through a methodical, proactive process. The key for success in creating this process is to make sure it is driven by data.
(Note: This article was originally published on User Interface Engineering’s web site.)
Posted in Articles - No Comments »
June 17th, 2008
By Christine Perfetti - August 15, 2007
Kim Goodwin is the General Manager and Vice President of Design at Cooper. The great folks at Cooper created an interaction design methodology known as Goal-Directed Design. Their methodology identifies the goals and behaviors of users and directly translates them into the design. I recently had the chance to talk with Kim about her work.
Alan Cooper, the founder of Cooper, originally popularized the concept of personas. How does Cooper define a persona?
As personas have become more and more popular, many people have started to misuse them, so there are a ton of misperceptions out there about what personas really are.
A persona is a behavioral model. The most effective behavioral models are distilled from interview and observation data of real users into an archetypal description of how a particular type of person behaves and what their goals are.
The two essential components of a persona are the persona’s behaviors and goals. For any product or service, design teams have to create multiple personas that represent the range of likely behaviors and goals.
Some design teams base their personas on a real person instead of a fictitious archetype. What do you think of this approach?
Certainly there are some real people who are very similar to a persona the design team may create, but it’s a dangerous approach because real humans are idiosyncratic. For example, any individual user might hate the color blue or have some other random opinions that aren’t necessarily representative of a larger population.
One of the strength of personas is that they gloss over those little idiosyncratic things and really focus on the essence of what is common to this particular type of person. That’s one of the reasons why we rely on personas instead of real users–they are more representative.
When creating personas, you focus on up-front user research. Can you describe what this research involves?
At Cooper, we rely heavily on ethnographic techniques. We focus on the context of use, looking at how people behave in their environment, whether that’s an office or an operating room, or on the train somewhere. We can be investigating an existing version of a product, a competitive product, or even a paper and sneaker-net version of something that doesn’t exist in a digital form yet.
In our research, we focus on uncovering how people use their existing tools and what mental models people have about their tasks. We also investigate how people currently accomplish their tasks, where they experience frustrations, and where there are opportunities for improvement.
Personas are really only one aspect of Cooper’s rigorous interaction design methodology, Goal-Directed Design. Can you describe the steps involved in your methodology?
We call our methodology “Goal-Directed” because it focuses on accomplishing goals. It’s important to note that these are not only the persona’s goals but also the business’s goals. If design teams only focus on the persona’s goals to the exclusion of making profits, the product won’t be successful.
In an ideal world, we start out conducting user research in contexts where people use the product or the service. Then, we create personas based on usage patterns and the sets of goals we’ve observed. Typically, we have only a small set of personas. For a consumer web site, we have about six personas. For a complex enterprise application where there are a lot of complicated interrelated roles, we may need 20+ personas.
Next, we use the personas to drive scenarios. The persona-scenario pairing is critical because if you just have a persona, it’s like having a really interesting character, but no story to tell. Just like you won’t have a good book or movie unless there’s a story, design teams won’t get a good product design with just a persona. They have to say to themselves, “In this situation, how would this persona ideally like to experience this interaction?” and “How does that look?”
By pairing the personas with the scenarios, we gather requirements, and from those requirements, we create design solutions. To create the initial design solutions, we use scenarios to drive what functionality is available and what’s paired together. We also use design patterns and principles to help us figure out how to concretely represent that functionality. These steps all come together in what we call the interaction framework, which is the initial sketch of the design. For example, how many screens does the solution have, how do users move around the interface, and what kinds of big things does it accomplish?
From there, we again use scenarios at increasing levels of detail to refine the design and constantly iterate it all the way down to the pixel level. Of course, there’s plenty of developer collaboration, and visual design in the mix, but this is our methodology in a nutshell.
Should every feature in a Goal-Directed Design be tied to user research?
Yes and no. Every feature should be tied to research in some fashion. Most of it’s going to come from user research, but occasionally a feature is driven by something like a regulatory requirement in healthcare that may not have anything to do with a user goal but the design team must include it or the product won’t ship. So, every feature and function in the design is traceable back to something, mostly user research.
Many of our clients see the importance of creating personas. However, some teams have very limited time and resources to go out into the field and talk with their users. Do you have any recommendations for dealing with these constraints?
People often have misconceptions about the word “research.” When you start using the word “research”, people automatically start thinking months and millions of dollars, and that’s really not the case.
Design teams can conduct rigorous research for a simple product in just a matter of a few days. In many cases, teams can conduct research for a really complicated enterprise application in under a month. It’s a matter of making sure people understand the scale of effort.
When we have a really tight time frame at Cooper, our approach is to look to friends and acquaintances and say, “Can we find a handful of people who are at least something like the type of user we’re trying to recruit?”
It’s not ideal, and obviously you’ll have less confidence in what you’re designing, but three or four perspectives different from our own can still help us to see the world in different ways.
Of course, designers won’t want to base a multi-million dollar development effort on three casual user interviews with friends of friends. Even so, if the time frame is compressed and it’s the only choice the team has, it’s better than not doing anything.
In Cooper’s Goal-Directed Design methodology, what is the team composition?
Our typical design team consists of two people who form the core of the team. One is a full-time Interaction Designer who works on that project and only that project. We also have the Design Communicator, another member of the team that works full-time on the project. We invented the Design Communicator role at Cooper over a decade ago. At first we thought, “Let’s hire technical writers to document what the interaction designers are coming up with.” However, we found that if you hire the right kind of technical writers, they don’t just write down what you say, they say, “Well, why is that good? And why would you do it that way?”
We’ve found that, by hiring the right people, they were naturally inclined to help us refine the design and clarify it. They were very rigorous in making sure it was fully articulated and that we had considered everything.
The Design Communicator role has really become a design partner, and so the interaction designer and design communicator are sort of like two halves of a brain. It’s like Scully and Mulder on “The X-Files.” One says, “Well, it ought to be this way!” and the other one says, “OK, I’m not sure I buy that.”
(Note: This article was originally published on User Interface Engineering’s web site.)
Posted in Articles - No Comments »
June 17th, 2008
By Christine Perfetti - June 8, 2006
Gerry McGovern is a world renowned content-management expert and author of the books, “Content Critical” and “The Web Content Style Guide.” User Interface Engineering’s Christine Perfetti recently talked with Gerry about the importance of a customer-centric approach to design. Here is what Gerry had to say about his experiences.
Christine Perfetti: Over the past several years, you’ve been traveling the world sharing content management and information architecture best practices. Have you come to any new insights?
I’ve seen that customer focus is the essence of the web economy. The web changes the dynamics of the relationship between the organization and the customer. The customer is more empowered, more in control.
Most organizations aren’t focusing enough on the customer. Their marketing material might talk about how important the customer is, but the culture of most companies is organization-centric—they focus on themselves. The problem with this approach is that organization-centric websites fail. The customer-centric websites are the ones that succeed.
What changes have you seen on the Web since you first started talking about it in 1994?
I’ve experienced three major phases for the Web: Infrastructure, Architecture, and Content.
In the early days (1994-1999), the major focus was on Infrastructure. Just getting everything technically working was not that easy. There was a big fascination with technology and a belief that all you needed to do was get the right technology and everything else would fit into place.
Around 1999, a greater emphasis on Architecture began to emerge. People had experimented a lot with layout and navigation, and there was a sense that it was time to begin formulating some rules. Large organizations, which often had totally different designs for their many websites, began to develop best practices and standardize.
In 2003, content began to be recognized as important. The infrastructure and architecture challenges had matured somewhat. There was more time to think about content. Over the years, it surprised me how little respect content had within most organizations. It was seen as a menial, trivial task. Print content was copied and just put up on the Web. From 2003 onwards, the better organizations recognized that there was indeed such a thing as web content, and that if it was professionally managed it became the engine of value for the website.
What led to this increased emphasis on a site’s content over the technology?
More and more web teams are realizing that infrastructure and architecture gets you on the pitch, but it is content that will win you the game.
Content makes the sale, delivers the service and builds the brand. The architecture is the container of the website, but content—well, it’s the content in the container. We don’t buy from iTunes because of its architecture; we buy because of its music. Great information architecture is invisible so that the content can shine through.
You’ve stated that only a small percentage (approximately 10%) of content really makes any difference online. Does this mean that the bulk of content is a waste?
Essentially, yes. Most organizations face two challenges when it comes to content: data management and content management. Organizations produce huge quantities of content/data that needs to be stored for legal and other reasons. Nobody’s interested in this sort of stuff, unless in exceptional circumstances.
Then, you’ve got a small quantity of content that the customer wants—that will help make the sale, deliver the service, and build the brand. This is what I call the killer web content. The other stuff is the filler web content. If you mix the two together, the filler smothers the killer. The job of a web manager is to identify the killer web content.
What are the first steps an organization should consider when they need to get their site’s content under control?
Stop thinking that they’re the center of the universe, and that any sane human being will be interested in reading their press releases and all the other self-congratulatory waffle they publish. Start thinking about how they can create a website that can serve their customers better. (This is as true for an intranet as a public website, where the customer is a staff member.)
Being customer focused is not some ‘nice thing to do.’ Customer focus is about hard-edged business. Customers are hugely impatient on the Web. They don’t need to hang around a website that is not directly focused on them. Customer focus is the beginning, middle and end of a successful web strategy.
In your book “Content Critical” you describe the web as a medium for publishing content and recommend that designers view their site as a publication. Given this, when hiring, what roles should an organization seek out for a successful web design team?
The number one skill that every web team should have is the ability and desire to relentlessly focus on the needs of the customer. Web teams must enjoy being around the customer, they must be stimulated by thinking of the customer. You have those skills and everything else fits into place.
The number one skill of an editor is not the ability to write. There are many people who are technically good writers but their content is not engaging. The editor must know their reader/customer inside out. They must also have empathy for their reader—be able to think like them, feel like them.
How do you recommend teams go about testing the effectiveness of their site’s content?
An emphasis on usability practices is critical. Get in front of the customer. Watch how they use your product. But, of course, there is nothing new about this approach. It’s how Ray Kroc built McDonald’s. It’s how Sam Walton built WalMart.
A testing technique I have been developing over the last five years is called Customer Carewords. It’s a voting technique that allows customers to vote for the words that mean most to them. Words are what drive actions on a website and if you identify the exact right customer words you’re going to achieve more success.
Has the emergence of RSS feeds changed the way readers interact with web content?
RSS is just another reflection of the empowered customer. It is the customer controlling how and when they want to receive the content. However, I have not noticed a substantial uptake in RSS.
The same with blogging, which is also about empowerment—people finding their own voice. Blogging is just one more sign of an empowered, intelligent, engaged society. No longer is the customer to be always talked at and told what to do. (This is not to say that mass marketing and advertising does not have a role for certain products and services.)
Do new design approaches, such as Rich Internet Applications and AJAX, change the way teams design for content?
I’m afraid I’m no expert on technical issues. Without technology we’d still be living in caves, but there is always a danger that technology becomes the end and not the means.
I tested a series of headings and summaries with 2,000 people in 12 countries. Some summaries and headings never got a single vote. These extremely poor performing headings and summaries had one word in common: technology.
I’m wary of web teams that can’t stop talking about the technology. Morning, noon and night, it is the customer you should be thinking and talking about.
Do you have any recommendations for design teams deaing with organizing content when the site has many different users with divergent needs?
Great websites tend to have a very clear purpose. Like Google, Skype, Netflix or iTunes, they tend to do a few things really well. Websites that are trying to talk to multiple customer groups with multiple needs are by their nature very complex. Simplicity is the foundation stone upon which the self-service model and the convenience society is built.
You might have a divergent customer base but there might be a few tasks that are common to all these customers. Many different types of customers go to an airline website but most of them want to book a cheap flight.
Most sites that have divergent customers with divergent needs probably should either seriously scale back or shut down. It is always a big danger sign when a web team claims all its customers are so different. It generally reflects a lack of management. Any website that tries to serve every customer and every need will do nothing well.
Posted in Articles - No Comments »
June 17th, 2008
By Christine Perfetti - April 24, 2007
For more than seven years, I’ve been teaching and coaching design teams on how to conduct usability tests and gather user feedback early on in the development process. One of the questions that comes up time and time again from clients is, “How can we get buy-in for usability tests from management and other team members?”
Through our own research at UIE, and in our ongoing discussions with expert usability practitioners, we’ve identified several proven techniques for getting stakeholders onboard.
1. Start Testing Right Away
Start testing. Start doing it right away. We’ve found there isn’t any one experience more beneficial to design teams than running a usability test. I’m still amazed by how quickly development team members recognize the benefits of usability testing once they’ve actually seen it in action.
When I’m teaching courses on usability testing, I’ve found that no amount of lecturing about the benefits of testing gets development teams onboard and past their skepticism. Instead, people only truly comprehend the power of testing once they’ve observed a user interacting with a design.
If you’re struggling to communicate the value of testing to your management or fellow team members, stop explaining the benefits and start demonstrating them. I’ve yet to see a test where the design team fails to gather some new piece of valuable information about the users’ needs.
When development teams start watching users interact with their designs, they’ll typically see two possible outcomes, both positive. In some instances, usability tests confirm the team’s existing beliefs about how users will use their products. But, in the much more common outcome, teams observe users experiencing problems with the design and identify gaping holes in their assumptions.
2. Debunk the Myth that Usability Testing Is a Big Production
One of the biggest obstacles design teams face when trying to sell testing is the perception that usability tests need to be a huge production.
The best way to tackle this resistance is by debunking the myth that testing has to be a big deal. Usability testing isn’t rocket science. The organizations that do the best job of incorporating usability tests into their existing process understand that testing is not a big deal.
The best organizations make usability testing a part of their everyday culture. To convince management that testing doesn’t need to be a huge production, we recommend design teams start simple. You can start by testing 3-5 users and disseminate that information throughout your organization.
Other experts agree. In an interview I conducted in 2002 with Rolf Molich, one of the world’s most respected usability practitioners, Rolf suggests:
“If your goal is to “sell” usability in your organization, then I believe 3-4 users will be sufficient. Much more important than the number of users is the sensible involvement of your project team in the test process and proper consensus-building after the test.”
Many teams are resistant to usability testing because of the belief that they need to spend money on state-of-the-art usability labs, or they are concerned they won’t find the right users. Again, our recommendation is to start simple: Test on a computer in your office cubicle and start testing with someone, even a co-worker, to begin gathering data. A usability test doesn’t need to be perfect. It just needs to happen.
3. Start Testing Early in the Process.
Many organizations are concerned that testing will disrupt project timelines because it may necessitate major design changes before launch.
However, time and time again, we find that design teams actually save time (and money) when they start testing at the beginning of a project. By finding usability problems very early on, teams prevent themselves from going in the wrong direction, leading to wasted time and resources.
The most successful teams have learned that the best way to create usable designs is to make informed decisions from the beginning of a project. They view testing as a technique to gather information to create great designs in a more timely and efficient way.
One of the best techniques for getting early feedback on a design is paper prototyping. Using common office supplies, development teams can build a working prototype of a design in a matter of days. We recommend that teams use this technique in the first few weeks of development. By identifying usability problems early, stakeholders very often see that the technique saves valuable time in the long run.
4. Involve Management and Stakeholders
To get buy-in from team members and management, it’s essential to keep them involved. On every project, we suggest that stakeholders sit and observe at least one usability test. This will give team members the opportunity to observe first-hand the information gathered from tests.
In an interview I conducted with Ginny Redish, a world-renowned usability expert and co-author of the book, “A Practical Guide to Usability Testing,” Ginny stressed the importance of involving team members:
“Involve them. Be a team together. Invite them to observe. If you have an observation room for them, put out candy or other food as an incentive for them to come.
Have someone in the observation room with them to monitor their conversation and, if necessary, join in to keep them from jumping to conclusions too quickly. If you do not have an observation room, set up a schedule so that you only have a few observers in the room at one time. Give them brief instructions on how to behave.
Involve them in a debriefing right after the testing; involve them in helping to find good solutions to the problems.”
Sarah Bloomer and Susan Wolfe, two premier usability practitioners also agree with Ginny. In an interview I conducted with them last year, they stated:
“It’s really essential that usability professionals collaborate with the product teams and forge strong relationships with these groups. The key is to treat these other groups as stakeholders in the process and keep them involved as the project evolves. That’s the only way to get their buy-in for the results. It never works to simply hand it to them at the end and say, “here you go!”.
Stakeholders need to have an ongoing role in the project beyond simply providing input at key milestones. We recommend holding stakeholder workshops to collect their broad range of wants and needs. This allows you to identify the business drivers that can be balanced with those of users.
Another approach is to teach the stakeholders some useful User-Centered Design techniques that they can use themselves. For example, at the MathWorks, paper prototyping has proven to be one of their most effective tools. Because of this, the usability professionals train other members of the development team to use the technique. As a result, it’s been incorporated into their development process.”
5. Identify Your Organization’s Champions and Address Their Needs
Finally, one of the best ways to get buy-in is to identify which members of your organization will benefit most from usability tests and recruit them as your Champions, assisting to rally other members of the organization
In my interview with Sarah Bloomer and Susan Wolfe, they recommend that usability professionals and design team members avoid evangelizing about the benefits of usability practices:
“When trying to get buy-in from team members, teams need to avoid the role of evangelist for user-centered design.
For example, let’s just consider the IT folks within an organization. If they’re not familiar with our field, their first reaction will be that this “UX stuff” will delay their projects and hamper their ability to meet their deadlines.
These concerns are real because their performance isn’t measured in terms of the success of the user interface. Instead, members of the usability or UX team need to demonstrate the value of in terms of outcomes that matter to IT, such as less rework and the ability to develop reusable code. Each stakeholder group has their own set of priorities, which needs to be understood and addressed.”
In our work, we’ve seen that it’s essential to focus on demonstrating to your organization’s Champions how usability tests will address their specific needs.
(Note: This article was originally published on User Interface Engineering’s web site.)
Posted in Articles - No Comments »
June 17th, 2008
By Christine Perfetti - September 09, 2004
If you’ve ever done any usability testing, then you’ve been affected by Ginny Redish’s pioneering work. Ginny is a world-renowned usability expert and co-author of the books, “A Practical Guide to Usability Testing” and “User and Task Analysis for Interface Design”.
While preparing for Ginny’s full-day seminar at the User Interface 9 Conference, UIE’s Christine Perfetti had the opportunity to ask Ginny about her thoughts on the best practices surrounding usability testing. Here is what Ginny had to say about her experiences.
UIE: How has usability testing evolved as a technique over the past 10 years?
Ginny: More and more companies and government agencies are doing usability testing. And, they are doing more of it, starting earlier in the process, and testing iteratively. Over the past 10 years, usability testing has become more integrated into the development process, especially in web design. As a result, we are doing more informal, formative, diagnostic usability testing now than ever before.
Has your philosophy changed at all since you first began usability testing?
No – and yes. My philosophy of usability testing has always been that it is the best way to find out how well a draft or prototype or product is doing for its users. I’ve always believed that usability is about helping designers and developers create products where users can quickly and easily find what they need and understand what they find. I’ve always believed that usability specialists should be part of the product team from the beginning and should do testing with the team – not as the “usability police.”
But my philosophy of how to do usability testing has also expanded over the last 20 years. When I started out, almost all usability testing was done in a formal lab with a very “hands off” approach to interactions between participant and facilitator. Today, the line between usability testing and field studies has blurred quite a bit. Typically, today, I sit with the participant. Depending on the stage the product is in, I may engage in much more dialogue than I did when I started out. I’ve done usability testing in conference rooms and cubicles; I even did one this summer in an airport hangar.
Your work focuses on helping usability practitioners to improve their usability testing skills. What recommendations would you give first-timers? Are there any common mistakes you see with design teams just starting out with usability testing?
Some aspects that I find design teams often need help with are:
- Thinking about the issues – what you want to learn from the usability test
- Writing good scenarios – that test the web site or product without giving away too much
- Facilitating comfortably – knowing when to talk and when not to, how to ask neutral questions, how to keep participants thinking aloud
- Taking good notes without missing anything critical
- How to report results so that the right people act on the
What recommendations do you have for usability professionals who want to get their development team on board and excited about usability testing?
Involve them. Be a team together. Invite them to observe. If you have an observation room for them, put out candy or other food as an incentive for them to come.
Have someone in the observation room with them to monitor their conversation and, if necessary, join in to keep them from jumping to conclusions too quickly. If you do not have an observation room, set up a schedule so that you only have a few observers in the room at one time. Give them brief instructions on how to behave.
In either case, give them note-taking forms and a bit of instruction on how to use them. Involve them in a debriefing right after the testing; involve them in helping to find good solutions to the problems. Include some positives in the report.
In your experience, what are the typical costs associated with usability testing? What are the budgeting considerations?
I’m always surprised when people think usability testing costs a lot of money. Yes, it costs money – but it’s usually a tiny fraction of the money being spent on the technical side of a project. Whether these are internal costs – or hard dollars you give out to someone – depends on how much you can do in-house. The major costs for usability testing are
- Time for internal people (and a usability consultant if you need one) to plan, prepare, conduct, analyze, and report
- Recruiting and paying participants (You may find it is less expensive to use a recruiting firm than to have internal people spend lots of time on the phone trying to find the right people.)
- Facilities (if you rent a lab, for example) – but you can also do usability testing in a conference room or even at people’s desks
Many usability practitioners believe that five to eight users is enough to find the majority of usability problems on web sites. In your experience, how many users are enough for testing?
“How many users” is one of the great controversies in usability testing today. I think it’s a false controversy because there is no one answer. The number you need depends on the type of testing you are doing, where in the development process you are, how many different types of users the web site or other product has, and a few other factors. I have slides in the seminar that explain all the factors you need to consider to decide on how many users.
The type of usability testing that we’ll focus on during the seminar day is the type that most companies and agencies are doing – testing prototypes of web sites or documents or software in the design stages for the purpose of finding and fixing major problems. My experience over many, many years of that type of usability testing is that you’ll find the major problems with relatively few users (I usually say six to 12 ).
Many design teams attempt to use heuristic evaluations as a less expensive alternative to usability testing. Where are your thoughts on this method?
Usability testing is only one of many techniques in the usability professional’s toolbox. You can use a heuristic evaluation (having one or more experts review the product) to catch major and obvious flaws in a product – if there are any. If the developers are willing to accept the results, they can bring a better product to usability testing. However, a heuristic evaluation is not an alternative to usability testing. A heuristic evaluation is just a prediction of what users will do. Until you see the real users, you don’t know whether those predictions are right. Only usability testing shows you where the real problems are.
You recently conducted two usability studies with Mary Theofanos and colleagues at the National Cancer Institute researching how vision-impaired users interact with web sites. What were some of your key learnings from this research?
Our first study was with 16 blind users; our second study was with 10 low-vision users. In both, we were watching and listening as our participants used web sites.
From our first study, we came up with many specific guidelines to help web designers and developers really make sites accessible – not just meet the letter of the law. One seemingly simple but very important learning is that many blind users do not know what Skip Navigation means. They want to skip over “all that stuff” at the top of each page, but they don’t click on the link that would help them do that. You can help them by changing the name of the link – and I’ll talk about how well different names worked.
Our second study was even more interesting because we found that the needs of low-vision users are so diverse that simple solutions are not going to help everyone. Adding accessibility on after a web site has been developed is not working. We need a new paradigm for thinking about “experience equity” – making web sites work for everyone.
(Note: This article was originally published on User Interface Engineering’s web site.
Posted in Articles - No Comments »
June 17th, 2008
By Christine Perfetti - July 24, 2003
You may have never heard of Rolf Molich. Yet, if you’ve done any usability testing, design evaluations, or heuristic inspections, then you’ve been affected by his pioneering work.
Since entering the field in 1983, Rolf has produced some of the most impressive and forward-thinking research on effective discount usability techniques. Two of Rolf’s more renowned contributions include the co-invention of the Heuristic Inspection process with Jakob Nielsen and the more recent CUE (Comparative Usability Evaluation) studies.
The Heuristic Inspection approach turned the usability world on its head when Rolf and Jakob suggested that you could get value by having experts review interface designs. However, in recent years, Rolf has revisited his thinking on this method and now is questioning its effectiveness for all projects.
The CUE studies are the first of their kind. Usability practitioners from all over the world are asked to evaluate the same interface, using their standard practices. Rolf, along with Robin Jeffries and other collaborators (including the interface’s design teams), compared the different results, looking to see which practices were most effective at discovering and reporting usability practices.
The most famous study, CUE-2 had nine teams conduct usability tests of Microsoft’s Hotmail interface. More recently, CUE-4 had 18 evaluators (using both expert inspections and usability testing) looking at iHotelier’s Flash-based hotel reservation system.
While we’re still waiting for many of the results from the CUE-4 analysis, the CUE-2 study has changed the way we think about usability testing practice. (For example, many questions arose about how “scientific” usability practices really are when there wasn’t one problem that all nine teams reported.)
While preparing for Rolf’s full-day seminar at the User Interface 8 Conference, we had the opportunity to ask Rolf about some of his thoughts on the best practices surrounding usability testing. Here’s what we talked about:
UIE: Many critics of usability testing argue that usability testing can’t make up for a bad design. Do you agree that if a design team starts with a deeply flawed design, usability testing will diagnose many of the problems, but won’t necessarily point to a cure?
Rolf Molich: Alan Cooper has wisely said “If you want to create a beautifully cut diamond, you cannot begin with a lump of coal. No amount of chipping, chiseling, and sanding will turn that coal into a diamond.”
That said, I have helped a lot of my clients produce rather usable pieces of coal based on simple rules for how to write good error messages, re-phrase key messages, tune a local search engine, or make other kinds of quick-and-dirty last-minute changes.
Many usability practitioners believe that “eight users is enough” to find the majority of usability problems on web sites. In your experience, how many users is enough for testing?
It depends. The number of users needed for web-testing depends on the goal of the test. If you have no goal, then anything (including nothing) will do.
If your goal is to “sell” usability in your organization, then I believe 3-4 users will be sufficient. Much more important than the number of users is the sensible involvement of your project team in the test process and proper consensus-building after the test.
If you goal is to find catastrophic problems to drive an iterative development process, then 5-6 users are enough with the current state of the art.
However, if you want to find all usability problems in an interface, then a large number of users and facilitators will be necessary as shown by the CUE studies and UIE’s research. In the CUE-2 and CUE-4 studies tests with more than 50 users brought us large numbers of valid problems, but by far no exhaustive problem list.
Since you and Jakob first started promoting Heuristic Inspections, you’ve indicated you’re not as optimistic about the technique. Where are your thoughts on that particular method today?
Heuristic inspections are cheap, simple to explain, and deceptively simple to execute. However, I don’t use this method very often and I don’t recommend it to my clients. In my opinion, the idea that anyone can conduct a useful heuristic inspection after a crash course is rubbish. The results from my studies showed that inexperienced inspectors working on their own often produce disastrous amounts of “false alarms”.
Another problem is that heuristic inspection is based solely on opinions. No one has given me a good answer to the question that I’ve heard several times from disbelieving designers: “Why are your opinions better than mine?” I think that’s an excellent question, particularly knowing that users often prove me wrong whenever my heuristic predictions are put to a real usability test.
What prompted the CUE studies?
Curiosity and the need for solid data. With the CUE studies, I wanted to offer designers and usability practitioners a summary of current, state-of-the art usability testing practices. At the same time, I wanted to give the participating usability labs an opportunity to assess their strengths and weaknesses in the core practices of the usability profession.
What were the biggest surprises when you compared the processes and reports from each of the nine teams in CUE-2?
What surprised me most was that many of the tests did not fully live up to what I consider to be sound usability practices
In the CUE-2 study, nine teams tested the Hotmail website. Each team had three weeks to run the study, which included recruiting their own test participants and creating their own test tasks. We imposed as few restraints on the teams as possible to ensure that the teams did the tests exactly as they would have done if they had been ordinary client projects.
Many of the teams failed at creating professional test tasks that were realistic, frequently occurring tasks and free of hidden clues and jargon. Some teams also failed to distinguish between user data and personal opinions. Even more surprising was how unusable some of the usability teams’ reports were.
What elements were lacking in the test reports?
Above all, a good usability report must be usable. The main recommendations I give clients for creating a usable usability report are:
- Keep it short.
No more than approximately 50 comments and 30 pages. It’s the job of the good usability professional to limit the comments to the ones that are really important.
- Provide a one-page executive summary on page 2.
Include the top three positive comments and the top three problems. Four of the nine CUE-2 teams did not include an executive summary in their reports.
- Include positive findings.
The ideal ratio between positive findings and problems is 1:1, but I have to admit that I rarely do better than 1:3. The CUE-2 teams ranged from no positive comments at all to an excellent ratio of 7:10.
- Classify the comments.
Distinguish between disasters, serious problems, minor problems, positive findings, bugs, and suggestions for improving the interface. Three of the nine CUE-2 teams did not classify their comments at all. The remaining six each invented their own classification scheme.
Reports are of course useful, but even a perfect report is useless if it doesn’t cause beneficial changes to the user interface. For example, good communication with the development team through effective consensus building is far more important than a good test report.
(Rolf Molich offers a sample usability test report that attempts to follow these recommendations at http://www.dialogdesign.dk/utestreports.html)
In the CUE-2 study, there wasn’t a single usability problem that every team reported. The findings indicate a strong need for improvement in the usability testing process. Don’t you think your findings undermine the effectiveness of usability testing?
In my experience, usability testing is very effective for showing your colleagues what a usability problem looks like in their interface. But I think the study results indicate that usability testing is ineffective for finding all usability problems in an interface. Our results also indicate that it’s ineffective even for finding all the serious usability problems in an interface.
The CUE-2 teams reported 310 different usability problems. The most frequently reported problem was reported by seven of the nine teams. Only six problems were reported by more than half of the teams, while 232 problems (75%) were reported only once. Many of the problems that were classified as “serious” were only reported by a single team.
Even the tasks used by most or all teams produced very different results - around 70% of the findings for each of these common tasks were unique.
My main conclusion is that the assumption that all usability professionals use the same methods and get the same results in a usability test is plain wrong.
Given your findings, how can development teams confidently conclude they are changing the *right* problems on their web sites?
It’s very simple: They can’t be sure!
But if they are humble, listen to their critics, learn from their mistakes, avoid voodoo-methods, and use regular external coaching to catch bad habits, they may eventually detect so many real problems that it will drive the iteration forward in a useful way.
Given your results from the CUE studies, do you think usability testing will play a major role in creating usable web sites in the future?
Usability tests are spectacular. They are excellent for convincing skeptical colleagues that serious usability problems exist in an interface. But they are also inefficient and costly. We should use them mainly in an intermediate phase to establish trust with our colleagues, and then use much more cost-efficient preventive methods such as usable interface building blocks, reviews based on standards and proven guidelines, and contextual inquiry.
I hope that we will one day have huge libraries of generic interface building blocks that are thoroughly tested with real users and proven usable. I also hope that we will show how assembling such building blocks into full-blown websites by usability-conscious specialists will yield websites with a high degree of usability.
Thanks Rolf!
(All reports from the CUE-1 and CUE-2 studies are publicly available (http://www.dialogdesign.dk/cue.html).
Note: this article was originally published on User Interface Engineering’s web site.
Posted in Articles - No Comments »
