April 2016
« May    




Differences between User Testing and Heuristic Evaluation

January 12, 2013

What’s the best way to find usability problems in a user interface, have an expert look for the problems, or get users to stumble through a novel interface finding problems as they go?

The following picture shows Jakob Nielsen (from some time ago) carrying out usability engineering (


There are strengths and weaknesses of both user testing and heuristic evaluation. Experts have often “seen it all before” so it is easy for them to recognize new instances of old problems. On the other hand it is not always easy for them to put themselves in the mind of the user. User interactions during a user test may be a more realistic way to uncover user problems, but it is a walk up and use kind of use. Problems that might seem difficult when users first try out a user interface may quickly go away after the user gets a bit more experience with the interface.

You can also ask users to think aloud during user testing, and this type of “verbal protocol” can be a rich source of information about user mental models and why they are having problems. That type of rich insight is typically not available from a heuristic evaluation. So heuristic evaluation can never be a full replacement for user testing.

The heuristics in heuristic evaluation are general rules relating to issues (problem areas) that typically arise in user interfaces. General heuristics can be used (e.g., the ten heuristics recommended by Nielsen) as well as more specific heuristics (e.g., for web design, or for a particular domain. However, during the heuristic evaluation the evaluator is free to go outside the heuristics being used and consider more general user interface design principles in generating ideas about possible usability problems. Nielsen recommends that evaluators go through the interface at least twice. “The first pass would be intended to get a feel for the flow of the interaction and the general scope of the system. The second pass then allows the evaluator to focus on specific interface elements while knowing how they fit into the larger whole.”

It is possible to perform heuristic evaluation of user interfaces that exist on paper only and have not yet been implemented (Nielsen 1990). This makes heuristic evaluation suited for use early in the usability engineering lifecycle. However, user testing can be used in the same way, providing that both the user and the tester are willing to deal with a style of interaction where pressing a button on one piece of paper leads to another piece of paper with a different state of the user interface being represented on it as a response to the preceding user interaction.

If evaluators are experts in the domain, or it is a general interface that anyone should be able to walk up and use, then evaluators might be able to go through it doing heuristic evaluation without any further guidance. This is in contrast to user testing where scenarios are generally needed to motivate the user interactions that are likely to uncover usability problems. Scenarios are generally of the form “how would you carry out this task in this situation?” In some situations though, heuristic evaluation may also require the use of scenarios. Task analysis of users as they perform their work might be one way of generating realistic scenarios. In cases where a comparable interface does not yet exist, ethnographic observation might be useful to understand the tasks that users in the domain need to perform.

The output from both heuristic evaluation and user testing is a set of usability problems associated with the interface being evaluated. It would be nice if it was always obvious how to fix a problem once it was observed but this is not always the case. As an intermediate step experts might assess how violation of well known user interface design principles might have led to the problem. Where a user interface design principle violation can be identified it might provide useful guidance as to how to redesign the user interface so that the problem is eliminated.

As Nielsen has noted “because heuristic evaluation aims at explaining each observed usability problem with reference to established usability principles, it will often be fairly easy to generate a revised design according to the guidelines provided by the violated principle for good interactive systems. Also, many usability problems have fairly obvious fixes as soon as they have been identified.”

Some problems are inherently easier to fix then others. An example of such a problem would be having too many fonts being used in an interface (e.g., “it looks like a kidnap ransom demand letter”). In this case the obvious response would be to standardize on one or two fonts. Since detailed guidance about typography in interfaces is available this is one area where the transition from usability problem to redesign of the interface tends to be less problematic.

Heuristic evaluation has been promoted as a “discount” usability engineering method. In 1994 Nielsen cited a study where the cost of using heuristic evaluation was about $10,500 and the expected benefits were about $500,000 (Nielsen 1994). Ideally, one would like to uncover all the problems that can reasonably be uncovered, and to do this user testing is needed. Another argument for user testing is that is hard for experts to trace through all the possible interactions that a user might have with an interface. On the other hand, you don’t have to find every problem in iterative design, just enough to proceed to the next iteration. So discount methods like heuristic evaluation or user tests with just one or two users may be necessary in order to follow the IDEO design strategy of “failing often to succeed early”.


Nielsen, J. (1994). Heuristic evaluation. In Nielsen, J., and Mack, R.L. (Eds.), Usability Inspection Methods. John Wiley & Sons, New York, NY.

Posted in: Events | Tags: , , , , , , ,

Leave a Reply

Follow Us!

Latest Posts