Recommendations and Best Practices in Mobile Usability Testing

NOTE: Much of this content is based on two earlier blog entries I wrote here a while back in 2009, so I added some additional and more recent details to come up with this revised version…

Prescreening users

First, prescreen your test participants. You need to know that they actually are capable of using the mobile devices that they own before you test. If you don’t do this for your test subjects, you may get a participant who has no clue as to how they can use their device beyond the basic function they do every day. If you don’t prequalify, you will get some “noise” in your test data that may make the whole session with that person worthless.

Recruit people who use the device, not just people who “own one”

Sometimes you will encounter a participant who got through the screening process and has NO IDEA how to do anything with his mobile device other than make or receive calls. You won’t get a lot of valuable data from this person, because a good half of your time with them will be spent explaining how to do things with his phone. Make sure your screener has questions to prevent these type of recruits from showing up.

Understand usage and context

In the pretest conversation, get a sense of how people actually use their mobile devices. Asking them what they do with their phones allows you to get a better sense of the priorities and features the user needs and provides valuable context that can inform your design goals.

Capture Everything

I’m a big fan of Morae from TechSmith, an app designed to record and analyze usability tests, and the latest version supports two video sources. The setup is simple: Use two cameras, one that is on a mounted tripod pointed down at the device, and the second monitoring the participant. Morae captures both video signals and records them, as well as transmitting the video of the session to a note taker via your work network.

Get comfortable with the devices you are testing

Even if you are recruiting users that are familiar with and/or are “power users” of a particular type of phone, you need to spend time getting used to how the device works. You may encounter a person who is not accustomed to the eccentricities of how the phone operates in the space you are testing (for example, a downloadable app on a blackberry). Be ready to work through such bumps by knowing how to do

When possible, test no more than two devices in one round of testing

If you are going from one device to another, you tend to lose focus on what you should be doing which is observing how the participants uses the app you are testing – you will focus more on how the device works than you should. So, test two device types in each usability test round, and try and have all the participants of Device A test on Day one and all of Device B test on Day 2.

Remember to separate the device’s usability issues from the application

On some phones you are going to see some very painful interactions that the participant has to go through to accomplish the task you have laid out for him or her. More often than not that is caused by the awkward data entry mechanism that the phone has and NOT the app itself. Always remove the usability problems the device causes from issues observed with what you are testing. If you’d like to catalog those issues and send them to LG or Nokia, fine, but that’s NOT your priority.

Make sure what you are testing supports the phone’s native controls

A couple of show-stoppers came up in a previous test when the participant used the native buttons/controls while accessing the app (the keypad’s “Back” button was one example). The app was not able to support this and logged the users out. If you have at least one participant do this, odds are that many more will being doing this as well. Make sure to note this as a MAJOR issue that the dev team needs to fix.

Creating prototypes to test

You may be testing a new design instead of an existing mobile app or site. You can use any design tool to create a clickable prototype, and can create either an html “mini-site” or a clickable PDF. Here’s some links (there’s a lot more out there):

Omnigraffle: http://viget.com/inspire/how-to-create-prototypes-with-omnigraffle

Axure: http://axureland.com/axure_blog/entry/mobile_prototyping_with_axure_rp_6.0

Keynote: http://blog.amirkhella.com/2010/07/13/teaser-iphone-running-an-interactive-prototype-built-with-keynote/

Getting the prototypes on the phone

You can use PDF readers to open PDF prototypes. For iPhone, I’ve used Keynotopia and for Android I’ve used a couple of different apps (whose name escapes me right now). Some PDF readers for Android don’t “pick up” the jump to links in the PDF, so be sure to test the prototype on the device before bringing in users. You can either copy the files to the phone or if that is not possible you can access the files through Dropbox on the device.

Designing on the device itself

A couple of apps have come out over the past few years that allow people to design or test paper sketches on the device itself. One iPad app called Blueprint allows you to design iPad and iPhone apps and export the results to a special reader/viewer. Another app called POP allows you to photograph sketches and then embed links that users can use to jump to different pages and views.

Holding the device

I’ve used – and like – Mr. Tappy (http://www.mrtappy.com). It’s a very solid metal handle for mobile and tablet testing that holds and elevates a webcam above the device (you can use software like Morae to “flip” the image if you need to). It’s pricey (at $295) but it’s good.

Can the physical appearance of a moderator influence the results of usability testing?

UX

Yes, the physical “attractiveness” can influence the responses of test participants. If you appear well dressed, it may make the participant feel they have to be more formal and some may look to you as an authority figure, and either reactions can affect how they respond. If you dress slovenly then some participants may not take you or the exercise very seriously.

A personal example: I don’t consider myself attractive, but I do consider myself reasonably good looking. I used to dress very well when I facilitated tests (suit and tie) and at 6 foot 4 in a suit I can make quite an impression (think Penn Jillette).

Over a couple of tests I noticed a couple of the female participants (and one male) was being… Well, they were flirting with me. Being an unbiased facilitator I had to not reciprocate and I had to “throw out” one of the interviews because the person was… Well, she was pretty blatant.

I was flattered, but I also decided to have less of a “presence” in the sessions – the focus of the test obviously is what you are testing and what the participant does and says… Not on you. So now it’s downkey polo shirts and nondescript khaki pants, to “blend in” and not distract.

See question on Quora

What should an ideal User Experience Design lab have?

I’ve set up a couple of usability labs in my time, so I’ll give you some of my advice/lessons learned in that space.

First the practical – get lots of wall space. I mean LOTS of wall space. Almost all the experience designers I have worked with love to use wall space to throw up mood boards, draft concepts, sketches, etc. Also wall space is ideal for affinity exercises – and to that end get lots of stickies, markers and white poster paper. Optimally you would have rolls of poster paper that you can mount towards the ceiling and then use to capture notes or a place to keep your stickies when you do brainstorming or analysis sessions.

You may also want to get dedicated project rooms, so you can keep all your work on the walls without having to move it from place to place. I’ve had to do analysis of data that was gathered through ethnographic research over a week-long period and switching rooms during that exercise was really frustrating and disrupted our workflow.

Now, hardware: use laptops, not desktops for your designers. The portability supports team members better. If you want to explore with next gen devices, get a touch computer like the ones HP has. If you really want to explore the space, get an XBox and a Playstation 3 and play with their new gestural devices/games. They are also useful for unwinding after the work day is done – as a famous starship captain once said, “The more complex the mind the more the need for play.”

When it comes to usability testing, if you can set up a permanent lab, do so. The need for a two-way mirror is debatable, with all the teleconference software that exists. I personally think that building in wall or ceiling cameras and microphones is overkill – the participants know they are being recorded during a test, so why hide it? Also overkill, in my opine and experience, is eye-tracking software and hardware. A good facilitator or observer will be able to get the same or better insight than eye-tracking would… It may not have the level of specificity that the software would provide, but I think the data it provides is usually not very actionable.

If you are doing mobile design (and you should be) then get a variety of devices and hire someone who can develop in that space… There are ways to get designs on the devices without coding but you may want to have the option to have more interactive mockups to test with. Get a monopod to lock a camera on a desk that can point down and record users using the device during testing (the smaller the camera the better).

And when it comes to usability testing software, there’s a lot of great programs out there. The one I prefer is Morae, from Techsmith http://www.techsmith.com/morae.asp It’s Windows only but it does everything you need it to do: it captures the screen the user is using (or the video from an external camera, for when you are doing mobile usability testing) and the video of the participant (from a webcam) and also transmits the video via your local network to any other computer that has the Morae Observer software installed. It allows people to watch a test in near-real time from another room or even another building.

When it comes to design software, I love Omnigraffle for the Mac http://www.omnigroup.com/ to do mockups, flow diagrams, mind maps, whathaveyou. I also love Axure RP http://axure.com/ to do designs and it generates functional prototypes you can use for testing. It’s available for both Windows and OS X.

Finally, set up a lending library and invest in some of the best books in design – I’m sure there’s other questions and answers around what these may be on Quora, so I’d consult those for advice.

See question on Quora

Need to do some mobile usability testing? Here’s some more hints and tips…

In some previous posts I gave some advice on how to conduct a usability test of mobile applications. Well, after a recent series of tests had several setbacks caused by our old pal Murphy showing up (no, not Peter Weller’s character from Robocop, the guy who created the law named after him) I have some additional advice and tips for you

Get comfortable with the devices you are testing

Even if you are recruiting users that are familiar with and/or are “power users” of a particular type of phone, you need to spend time getting used to how the device works. You may encounter a person who are not accustomed to the eccentricities of how the phone operates in the space you are testing (for example, a downloadable app on a blackberry). Be ready to work through such bumps by knowing how to do

When possible, test no more than two devices in one round of testing

If you are going from one device to another, you tend to lose focus on what you should be doing which is observing how the participants uses the app you are testing – you will focus more on how the device works than you should. So, test two device types in each usability test round, and try and have all the participants of Device A test on Day one and all of Device B test on Day 2.

Recruit people who use the device, not just people who “own one”

Sometimes you will encounter a participant who got through the screening process and has NO IDEA how to do anything with his mobile device other than make or receive calls. You won’t get a lot of valuable data from this person, because a good half of your time with them will be spent explaining how to do things with his phone. Make sure your screener has questions to prevent these type of recruits from showing up.

Remember to separate the device’s usability issues from the application

On some phones you are going to see some very painful interactions that the participant has to go through to accomplish the task you have laid out for him or her. More often than not that is caused by the awkward data entry mechanism that the phone has and NOT the app itself. Always remove the usability problems the device causes from issues observed with what you are testing. If you’d like to catalog those issues and send them to LG or Nokia, fine, but that’s NOT your priority.

Make sure what your are testing supports the phone’s native controls

A couple of show-stoppers came up in a previous test when the participant used the native buttons/controls while accessing the app (the keypad’s “Back” button was one example). The app was not able to support this and logged the users out. If you have at least one participant do this, odds are that many more will being doing this as well. Make sure to note this as a MAJOR issue that the dev team needs to fix.