maintained by Lars Pind.
(updated June 19, 2001)
Usability testing is essential to a good user experience. How will you know it's good if you don't test it? Once you've worked on a UI for some time, regardless of in what capacity, you can't see it fresh anymore. You need outside eyes. It's an invaluable way of seeing what kinds of problems "real" users stumble on. After all, you're not designing for yourself. You're designing for your users.
In addition, this is a good way of testing requirements. It's generally hard for people to look at a list of functional requirements and determine whether it matches their needs. It's much easier to look at the interface, pretend to be using it, and then see what's wrong or missing.
Test early and often. Test whatever you have at each given step: Mock-ups, wireframes, prototypes, real implementation. And every time you change something after doing a round of tests, test it again. Make it, test it, fix it, test it again. Make sure that testing is so light-weight that you have the energy and time to test several times.
Getting representative users isn't terribly important. Most common usability problems can be caught by anyone who's not familiar with the software already, and who doesn't have any special disqualifications (such as being a programmer). If you're not testing usability, but, say, you're testing whether the functionality is relevant for the target audience, then you do need domain experts to test that. But one round of this might be enough. Not every round of tests is going to be like this. Just make sure you get people that are not familiar with the ACS, whenever possible, since we have to avoid ACS'isms as much as possible. For the price of a pizza often you can get fresh eyes.
Finally, keep in mind, that you're not trying to prove or disprove something. You're testing to inform your own judgment. We don't just let the users decide. In fact, the optimal situation is if they behave like we weren't there at all, exactly like if they were at work, using the software. We don't want them to stop and reflect, we just want to be that fly on the wall. We want to improve their normal work life, and in their normal work, reflecting on how the interface could be different won't help them.
You'll want to be two people conducting the test, that is, in addition to the test subjects. One will be the facilitator, instructing the user and directing the test, while the other will be the observer, making notes of the things that are discovered. If you have two rooms and a camcorder, sit the observer in the room next door. Otherwise, he'll be in the same room, but he should remain quiet.
Having three to four users per round of testing is optimal. If you test fewer, you catch too few problems, and you increase the risk that the things found don't generalize to the larger population. If you test more, you get too many repeats, and likely also more notes than you can manage. The effort will be better spent on another round of testing once you've fixed the problems found in this round.
Also, make sure you know what you're testing, and plan the test accordingly. Some examples include:
As a rule of thumb, give as little background information as possible. You want to see if people can make sense of the interface on their own.
The discussion below focuses on testing for usability problems. If you're testing something else, you'll have to improvise and adjust accordingly.
Make sure you run the test yourself first. You don't want to ask your users to do a task, only to find out there's a bug in the software so it can't be done. Try it out the day before!
Protect the self-esteem of the users. It's easy for them to feel stupid, and it's your job to make sure it doesn't happen. Explain carefully that you're not testing them, you're testing the software, and that if there's something they can't figure out, it's your fault, not theirs (but they must try anyway). Tell them they're helping you, and that you appreciate it. Tell them before the test and after the test, and remind them during the test if you need to. You know they're not stupid. Tell them.
Ask them not to click on anything until they've told you what they want to click on, why, and what they expect to happen. The observer is paying keen attention to this. You want to know what they expect to happen, so you can learn what expectations your interface is raising in their minds, and how your software may be misaligned with those expectations.
Ask them to think aloud. Focus on all the unnecessary thinking, resulting from confusing interface. "Hmm ... I wonder what that button does ... Where did they put search ... "Find"? Could that be the search? ... Where can I find the number of users in this group ...". Whenever you're unsure what they're thinking, ask them: "What are you looking at?", "What are you thinking?". If they're talking about "navigation", or "layout", or something general like that, ask them to be specific about what they're referring to, so you know what exactly they're reacting to.
What you're looking for is not their opinion on the interface, but the problems, frustrations, and behavior in real use. So you don't want them to use the interface, not reflect on it. If they start reflecting, try to get them back on track, by kindly reminding them of the task they're performing. Tell them that you'd very much like to hear their thoughts once the formal exercise is over.
In the same vein, do not give hints. Your real users won't have you around at work. Ask them what they think they should do, or what they'd do if you weren't there. The only exception is when the user is completely stuck, or the user is stuck on some problem you already know, where the fix is obvious. In that case, you may give a verbal hint. But do not, ever, touch the mouse or keyboard. Let the user do everything, Let them drive start to finish.
Again, you should feel free to improvise as needed. Sometimes, getting opinions make sense. Sometimes, giving a simple hint makes sense. If you can't get someone to be the observer, you'll have to fill that role yourself.
Make notes as soon as you're done with each user, whether you have an observer or not. When the next user starts testing, you'll have forgotten.
Look for whether users understand what things are, what they do, and how they achieve their task. Does the navigation work? Do they understand the structure? Can they tell where they are, and can they get to where they want?
Sometimes you'll get insights that are obvious, once they've been pointed out to you. Sometimes, the user just doesn't get something that's entirely obvious to you, such as a navigation aid at the top of a page. These are things that can be hard to see for someone who has worked with the interface for too long.
Remember that you're looking for the user's behavior and the reasons for that behavior, more than the user's opinions, even though most users will be more than happy to share their opionions, because they instinctively think that's what you want. And remember that the user will likely make a greater effort in the test situation than he or she would have on her own.
Summarize the problems you've found, and determine which ones need fixing, based on how bad they are, and how many of the test users experienced them. You may want to grade them according to these scales, i.e., severity and frequency. Be specific in describing the problems, i.e., what particular aspect of the navigation wasn't clear on what pages.
Focus your effort on the important problems. If the users were able to recover from a problem quickly and without help, spend your time on something else. If a problem seems very rare or obscure, move on. Use your judgment.
Examples of typical problems are: The user either doesn't get the concepts, or he does, but his understanding is different from that of the interface. Naming problems, where the interface uses a term that's unfamiliar or non-obvious to the user. Or the way you've split up or categorized things are out of sync with the way the user sees the world. Or the page might simply be visually confusing or cluttered.
Once you've listed the problems found, you'll need to find out how to fix them. Sometimes the solution will be obvious. Sometimes the tests have provided concrete hints at how the problem could be solved. And sometimes you just have to think hard.
Remember that you don't have to come up with the fix to end all fixes. All you have to do is find out what to try next. You're going to test the fix as well. First try to find the simplest thing that could possibly fix the problem, a minor tweak, then mock it up or implement it, and test it. But don't just blindly add more stuff, e.g., just adding more links to the things people couldn't find. You want to have less interface, but make it more useful. And be wary of feature requests. It's very common for people to ask for new things, e.g., "It would be nice if ...", or "I think I'd like to see the number of ...". Don't increase scope. Leave something for the next releas.e
Of course, if there's low-hanging fruit, take it. If something takes almost no effort to fix, but it would improve the experience significantly or even slightly, do it. But make sure you don't break the things that work. And remember to test it again.
But when it turns out that the problem really is in the software functionality, when the real culprit is bad requirements, face it as soon as possible. It might not be possible to fully fix this within the given deadlines, but at least look into what the problem is, what the fix is, and how you can best work around it for the current release. This is not at all an unusual thing to discover, and is one more reason that usability testing should be done early in the project.
Good luck with your testing. Please share your experiences on the bboards.
For more detailed coverage, chapter 6 of Jakob Nielsen's Usability Engineering is excellent and pretty thorough. The whole book is actually very fine.