From Sunday, May 22 to Friday, May 27, 2016 experts from Nielsen Norman Group, one of the leading consultancy companies in the field of user experience (UX), visited the city of Amsterdam for their periodical UX Conference. During that week I attended a full-day, in-depth session about usability testing. Ahead of the conference it was hard to pick one from no less than 24 courses because all of them looked very interesting. Course topics varied from customer journeys to information architecture and from UI principles to mobile UX. I picked usability testing because I know that it is a part of UX which I probably have the least experience in and for that reason I figured that it would be a valuable addition to my UX skillset.
There are plentiful reasons why anyone working in product development should have at least a basic understanding of what usability testing entails and why it is important. We are living in a time where people have come to expect products to be functional, flexible and reliable. There is high pressure on design and development teams to deliver rock-solid products that users can genuinely depend on. This could not be more true for today’s situation at Wolters Kluwer. Many of our customers rely on our products to help them do their work in the best possible way. We can not afford our products to fail. In order to create products that meet the high expectations of our customers we need to test them in the earliest stages of our development process.
We cannot depend on the designer’s first intuition about what users might need alone (and this is coming from a designer). We can significantly improve our designs by watching real customers interact with our products. As early as in the ideation phase we can already conduct field studies or a competitive analysis to validate assumptions. In early design iterations there is a possibility to test with a paper prototype or by card sorting to evaluate the information architecture. In the MVP stage a clickable prototype helps to uncover problems in the UI by getting first-hand feedback from real customers. It is important to note that in the earliest stages of the project you are usually validating problem-solution fit (does my solution solve my user’s problem?) whereas usability tests kick in later when you have validated your solution and there is a prototype available.
After a long day of taking notes, completing exercises, and processing course material I wrote down the three key takeaways from the usability testing course. (For post-size reasons I will give you the first one now and the other two in a second blog post.)
#1 – What a user DOES is the most important
From Sunday, May 22 to Friday, May 27, 2016 experts from Nielsen Norman Group, one of the leading consultancy companies in the field of user experience (UX), visited the city of Amsterdam for their periodical UX Conference. During that week I attended a full-day, in-depth session about usability testing. Ahead of the conference it was hard to pick one from no less than 24 courses because all of them looked very interesting. Course topics varied from customer journeys to information architecture and from UI principles to mobile UX. I picked usability testing because I know that it is a part of UX which I probably have the least experience in and for that reason I figured that it would be a valuable addition to my UX skillset.
There are plentiful reasons why anyone working in product development should have at least a basic understanding of what usability testing entails and why it is important. We are living in a time where people have come to expect products to be functional, flexible and reliable. There is high pressure on design and development teams to deliver rock-solid products that users can genuinely depend on. This could not be more true for today’s situation at Wolters Kluwer. Many of our customers rely on our products to help them do their work in the best possible way. We can not afford our products to fail. In order to create products that meet the high expectations of our customers we need to test them in the earliest stages of our development process.
We cannot depend on the designer’s first intuition about what users might need alone (and this is coming from a designer). We can significantly improve our designs by watching real customers interact with our products. As early as in the ideation phase we can already conduct field studies or a competitive analysis to validate assumptions. In early design iterations there is a possibility to test with a paper prototype or by card sorting to evaluate the information architecture. In the MVP stage a clickable prototype helps to uncover problems in the UI by getting first-hand feedback from real customers. It is important to note that in the earliest stages of the project you are usually validating problem-solution fit (does my solution solve my user’s problem?) whereas usability tests kick in later when you have validated your solution and there is a prototype available.
After a long day of taking notes, completing exercises, and processing course material I wrote down the three key takeaways from the usability testing course. (For post-size reasons I will give you the first one now and the other two in a second blog post.)
#1 – What a user DOES is the most important
My instructor of the day, Kara Pernice, Senior Vice President at Nielsen Norman Group, started off our day with a quote from cultural anthropologist Margaret Mead that immediately hit the nail on the head: “What people say, what people do, and what they say they do are entirely different things”. Watching users doing tasks on their device can yield very valuable insights, far more valuable than only asking people how they use something. Be aware that the path a designer hopes a user will take to fulfil a task (the happy path) hardly ever matches the path a user actually takes (the observed path). Look at how people work – the way they do their tasks – and look at the outcome. Completing a task is not enough. They way a user completes a task is more important. Does a user stick to something what works in a certain way, also known as ‘momentum behavior’? With what difficulty did a user complete the task? You want to test with existing and new users to compare your findings.A typical qualitative usability test is done with 5 to 10 users where you observe user behaviors and draw insights from your observations. Testing with only 5 users can uncover 85% of all the usability issues. Unlike quantitative testing – where many (20 or more) users are tested – qualitative tests don’t give you definitive, scientific ‘proof’ nor will you be able to establish statistical validity, which is fine because it’s not the goal of most usability tests. The goal of qualitative usability testing is to establish that there is an issue with your product, you uncover why there is a problem and you iterate your design from there. On average, as stated by Jakob Nielsen, user performance or productivity increases a whopping 112% by conducting iterative testing and design.
— PART II —
#2 – Determine your goals and what you want to learn
It is important to decide upfront what to test and what you want to get out of your tests. You can’t really conduct your study before defining your goals. As a team ask yourself what are the top things (up to ten) a user needs to do to be successful. You marked these things as the most important so they need to work. You can define the top tasks by doing a survey, by talking to product management or by analyzing your metrics. When defining your top tasks involve team members throughout the organization to create support. Anyone who has a mandate to say “no” or “let’s change the design” you want to involve. The earlier you involve them the better. This prevents creating an ‘us versus them’ environment that could harm the project.
You also want to make sure you have defined who the target user is so no one will say “this one is stupid”. Examples of user groups are novice users, domain experts, seniors, children and so on. Focus your test on a certain user group and limit the scope to specific areas of features. You could decide to limit the scope for each round of testing based on the importance of a (new) feature to the business or based on an area of the product you know causes problems.
#3 – Tasks impact what and how much you will learn
Writing tasks is, according to Kara Pernice, as much an art as it is a science. A task impacts what and how much you will learn. When writing tasks keep in mind that discoverability, findability and usability are the trifecta of usability testing. Think about what you want to learn about these three things. If you want to learn about discoverability of a feature, make sure you don’t tell your user about it but rather give a small hint for the need for the feature. When testing for findability you can tell a feature exists (without giving the exact name) and you observe if a user is able to find it. Once users are using it, see what works and what doesn’t and, more importantly, find out why.
Prioritize your tasks and give your users the most important ones first in case you run out of time. You could consider giving your users easy tasks first to warm them up and to make them feel good about something but consider the time limit and always pilot your tasks with a colleague. Make sure you realize that when you give people tasks you are directing them in a huge way. It could, therefore, be wise to start open-ended and hone in later especially if you want to learn more about discoverability. Try to take a step back first and start off in a more exploratory way if possible.
While completing tasks it is very important that your users think out loud. If they read anything –on a screen or on paper – ask them to read it out load. This help you to understand what your users are seeing, thinking and experiencing. After a user completes a task there should be an opportunity for post-task questions. Give your users the possibility to speak out on anything that might be important. Ask if a user has any comments about the activity they just worked on and ask if there was anything particularly easy or difficult about the task. At the end of a test session there should always be some time left for questions you wrote down about specific things the user was doing. It is for that reason that I leave you with the most important tip of all: take notes!
These two blog posts sum up the larger part of what was discussed and what I learned at the usability testing course. I hope I was able to convey why we need usability testing and how we could go about doing these tests in an organized and efficient manner. If there are any particular questions about usability testing or if there is something you want to contribute based on your own experiences, please send me an email or add a comment below.