June 17th, 2008
On your site, the content page is the user’s most frequent final destination. This page contains the information the user came to the site to find. Sites often have hundreds, if not thousands (and in some cases, millions) of these critical pages.
How can design teams be confident their content pages are understandable to users? How does a team ensure they’ve designed content pages that communicate the essential information effectively?
A simple usability testing technique can help design teams quickly measure how a content page performs with users. It’s called the 5-Second Test. In case you missed it, a few years back, I wrote an article about the 5-second test methodology, outlining how to set up these types of tests and when they are most effective.
Posted in Articles, usability testing - No Comments »
June 11th, 2008
By Christine Perfetti and Lori Landesman — June 18, 2001
One of the most popular questions we hear from web designers and usability professionals is: “How many users is enough when conducting usability tests?” Until recently, we had believed — and told our clients — that it wasn’t usually necessary to test with more than eight users. We based our recommendation on the widely held theory that eight users will detect almost all of your web site or software usability problems.
According to a paper published by Robert Virzi (1992), 80% of all usability problems can be found with four or five users. Additionally, the first few users will most likely detect the biggest usability problems. Jakob Nielsen and Thomas Landauer (1993) provided further support for this theory when they found that the first five users will uncover about 70% of the major usability problems and the next few users will find nearly all the remaining problems, up to 85% or so.
In a recent study, we decided to put the widely held belief that “eight users is enough” to the test. We conducted usability tests on an e-commerce web site using a very straightforward task: buying a CD from an online music store. We chose users who had a history of purchasing music online. We asked these users to make a shopping list of CDs they wanted to buy and gave them money to spend on these items. Based on previous findings, we expected to detect 85% of the site’s usability problems with the first eight users. We also suspected that all of the serious obstacles would be evident early. Additionally, after each progressive test, we expected to see an increase in repeat usability problems.
When we tested the site with 18 users, we identified 247 total obstacles-to-purchase. Contrary to our expectations, we saw new usability problems throughout the testing sessions. In fact, we saw more than five new obstacles for each user we tested.
Equally important, we found many serious problems for the first time with some of our later users. What was even more surprising to us was that repeat usability problems did not increase as testing progressed.
These findings clearly undermine the belief that five users will be enough to catch nearly 85% of the usability problems on a web site. In our tests, we found only 35% of all usability problems after the first five users. We estimated over 600 total problems on this particular online music site. Based on this estimate, it would have taken us 90 tests to discover them all!
When we explored the difference between our results and previous findings, the disparity started to make sense to us. While we had tested users on an e-commerce site, Virzi and others had tested users on software products. Today’s web sites, particularly e-commerce sites, can be more complex than standard software products that often confine users to a very limited set of activities. Web tasks are also vastly more complex than those users have with most software applications. For example, our tests asked users to complete shopping tasks. No two users looked for the same product and no two users approached the site in the same way. The tasks were dependent on individual user characteristics and interests. Because of the increased complexity of web sites, it’s understandable that more users are needed to detect the majority of usability problems.
These findings may sound daunting, but we’re not really advocating that you plan to bring in 90 users for your next major usability study. Rather, if you’re working on a large e-commerce site — or any web site at all — the usability of your site would likely benefit from ongoing testing. Instead of thinking of usability testing as a discrete activity that takes place every 6 months and involves six, eight or twelve users, think about the advantages of ongoing usability testing, bringing in a user or two every week.
With this kind of plan, you’ll see over 20 users in six months. With more users testing your site, you’ll get more feedback, find more problems, and have more data, but there may be some less obvious advantages as well. When a design team gets into the mindset of regular testing, they can try out new ideas and find out whether these work without making the live site the testing ground. Because web sites undergo many incremental, seemingly small design changes in between drastic redesigns, there’s always new fodder for testing.
Also, the constant exposure to users can benefit the whole team, which, with regular testing, will be less estranged from their user base. Understanding who uses the site, what language is important to them, what they’re trying to accomplish, and whether they can do it benefits the team, the site, and the company.
Note: This article was originally published on the User Interface Engineering web site.
Posted in Articles - No Comments »
June 11th, 2008
By Christine Perfetti — June 9, 2005
On your site, the content page is the user’s most frequent final destination. This page contains the information the user came to the site to find. Sites often have hundreds, if not thousands (and in some cases, millions) of these critical pages.
How can design teams be confident their content pages are understandable to users? How does a team ensure they’ve designed content pages that communicate the essential information effectively?
Designers constantly struggle with the task of creating successful content pages. For example, RedCross.org’s designers need their users to understand all the options for donations. When the world faces a major disaster like the South Asia tsunami, the designers want users to realize the flexibility they have to help.
A simple usability testing technique can help design teams quickly measure how a content page performs with users. We call it the 5-Second Test.
As the name suggests, the 5-Second Test involves showing users a single content page for a quick 5 seconds to gather their initial impressions. Five seconds may not seem like a lot of time, but users make important judgments in the first moments they visit a page. This technique unveils how those judgments turn out, giving the team insight into some essential information about the page. Using this technique, we’ve found the information we’ve gathered essential for making huge improvements to our clients’ sites.
The 5-Second Test Method
If you’ve conducted a usability test in the past, it’s very simple to put together a 5-Second Test. It has a similar structure to traditional usability tests—you’ll have users, tasks, and the site you’re testing. The difference is in the specific protocol for running the test.
Most content pages have a distinct primary purpose. In the case of RedCross.org’s Donation content page, the primary purpose was to communicate to users all of their contribution options. This is a high-risk page for the Red Cross. If it fails, they risked losing hundreds of thousands of dollars of essential contributions. (Figure 1 shows the Red Cross Donation page we tested.)
Figure 1: The Red Cross.org’s Donation content page needs to communicate to users all of their contribution options.
As we often do in other types of usability tests, we start by giving users a focused task. For the Donation page, we gave users a simple task:
“You’re ready to donate to the Red Cross organization. But you’re unsure of what kind of donation to make. What are your donation options?”
Next, before we show the user our page, we tell them we’ll only display it for 5 seconds. We ask them to try to remember everything they see in this short period.
Once the user views the entire page for 5 seconds, we remove it by either covering it up or switching to another window. Then, we ask them to write down everything they remember about the page. When they finish jotting down their recollections, we ask two useful questions to assess whether users accomplished the task. For the Donations page, we’d ask, “What is the most important information on this page?” and “How would you go about donating to the Red Cross?”
By paying careful attention to users’ initial impressions, we can identify whether the content page is clear and concise. If the page is understandable, users will easily recall the critical content and accurately identify the page’s main purpose.
In our test of the Red Cross page, all users could tell us what content was most important—they all told us the purpose of the page was to generate donations. However, the page was surprisingly ineffective at communicating the donation options.
When prompted, users told us they knew they could donate money to the Red Cross. What they didn’t learn was that they could also donate stock, clothes, and airline miles. Even though this information was readily available on the content page, no user realized there were other options. We learned the content page was not doing its job communicating all of the relevant information. To succeed, the team needed to highlight the different vehicles for donation better.
Recently, we repeated the test on a redesigned version of the Donation page. (Figure 2) This new page prominently lists all of the different ways of donating at the top of the screen. Testing the redesigned page, we found users readily recalled the complete set of donation options with only 5 seconds to look at the page.
Figure 2: With the redesigned version of the Donation content page, users readily recalled the complete set of donation options with only 5 seconds to view the page.
The Benefits of 5-Second Testing
Limiting the viewing time to 5 seconds, we get a valuable glimpse into what happens during the first moments a user sees a page. When we give users more than 5 seconds to study the page, we’ve found they start looking at the page more like a designer, noticing details they would normally miss or misinterpret.
Frequently, we’ll conduct 5-Second Tests with paper mock-ups or low-fidelity electronic prototypes, such as PDFs or Photoshop page renditions. We can test very early in the development cycle, long before the team builds a functional web site. Often, this early insight can help point out site-wide information design requirements, saving much redesign work down the road.
One of the 5-Second Test’s biggest advantages is how quick it is. When evaluating the Donation page, each user took only 10 minutes! Because this technique is quick and easy to implement, it is perfect to run in locations where we can gather many users at one time, such as trade shows, conferences, and the company cafeteria. We can gather large amounts of user data in a short time.
5-Second Tests Don’t Tell Us Everything
We’ve found this technique to be an essential part of our usability toolbox. However, it has limits to what it can tell us.
We’ve found the technique is best when we use it on pages designed with a single primary purpose. Home pages and major navigation pages don’t yield as valuable results, because they often serve many different tasks.
For example, the home page for RedCross.org serves the needs of donors, sponsors, volunteers, medical professionals, victims, and the press, each with their own set of tasks. Each of these different users would probably see different things on the page, depending on their context and immediate goals. Other techniques, such as traditional usability tests and inherent value tests would be better instruments for judging the effectiveness of this page.
A Quick Technique for Evaluating Content Pages
We’ve used 5-Second tests on any project where clients need to make quick improvements to their content pages. If you have a site or product where users tell you that the site’s content pages are cluttered or confusing, this may be just the right test for you. 5-Second Testing is one more technique in our user experience toolbox that gives teams the information they need to create successful designs.
(Note: This article was originally published on User Interface Engineering’s web site.)
Posted in Articles - No Comments »
