NOTE: Much of this content is based on two earlier blog entries I wrote here a while back in 2009, so I added some additional and more recent details to come up with this revised version…
First, prescreen your test participants. You need to know that they actually are capable of using the mobile devices that they own before you test. If you don’t do this for your test subjects, you may get a participant who has no clue as to how they can use their device beyond the basic function they do every day. If you don’t prequalify, you will get some “noise” in your test data that may make the whole session with that person worthless.
Recruit people who use the device, not just people who “own one”
Sometimes you will encounter a participant who got through the screening process and has NO IDEA how to do anything with his mobile device other than make or receive calls. You won’t get a lot of valuable data from this person, because a good half of your time with them will be spent explaining how to do things with his phone. Make sure your screener has questions to prevent these type of recruits from showing up.
Understand usage and context
In the pretest conversation, get a sense of how people actually use their mobile devices. Asking them what they do with their phones allows you to get a better sense of the priorities and features the user needs and provides valuable context that can inform your design goals.
I’m a big fan of Morae from TechSmith, an app designed to record and analyze usability tests, and the latest version supports two video sources. The setup is simple: Use two cameras, one that is on a mounted tripod pointed down at the device, and the second monitoring the participant. Morae captures both video signals and records them, as well as transmitting the video of the session to a note taker via your work network.
Get comfortable with the devices you are testing
Even if you are recruiting users that are familiar with and/or are “power users” of a particular type of phone, you need to spend time getting used to how the device works. You may encounter a person who is not accustomed to the eccentricities of how the phone operates in the space you are testing (for example, a downloadable app on a blackberry). Be ready to work through such bumps by knowing how to do
When possible, test no more than two devices in one round of testing
If you are going from one device to another, you tend to lose focus on what you should be doing which is observing how the participants uses the app you are testing – you will focus more on how the device works than you should. So, test two device types in each usability test round, and try and have all the participants of Device A test on Day one and all of Device B test on Day 2.
Remember to separate the device’s usability issues from the application
On some phones you are going to see some very painful interactions that the participant has to go through to accomplish the task you have laid out for him or her. More often than not that is caused by the awkward data entry mechanism that the phone has and NOT the app itself. Always remove the usability problems the device causes from issues observed with what you are testing. If you’d like to catalog those issues and send them to LG or Nokia, fine, but that’s NOT your priority.
Make sure what you are testing supports the phone’s native controls
A couple of show-stoppers came up in a previous test when the participant used the native buttons/controls while accessing the app (the keypad’s “Back” button was one example). The app was not able to support this and logged the users out. If you have at least one participant do this, odds are that many more will being doing this as well. Make sure to note this as a MAJOR issue that the dev team needs to fix.
Creating prototypes to test
You may be testing a new design instead of an existing mobile app or site. You can use any design tool to create a clickable prototype, and can create either an html “mini-site” or a clickable PDF. Here’s some links (there’s a lot more out there):
Getting the prototypes on the phone
You can use PDF readers to open PDF prototypes. For iPhone, I’ve used Keynotopia and for Android I’ve used a couple of different apps (whose name escapes me right now). Some PDF readers for Android don’t “pick up” the jump to links in the PDF, so be sure to test the prototype on the device before bringing in users. You can either copy the files to the phone or if that is not possible you can access the files through Dropbox on the device.
Designing on the device itself
A couple of apps have come out over the past few years that allow people to design or test paper sketches on the device itself. One iPad app called Blueprint allows you to design iPad and iPhone apps and export the results to a special reader/viewer. Another app called POP allows you to photograph sketches and then embed links that users can use to jump to different pages and views.
Holding the device
I’ve used – and like – Mr. Tappy (http://www.mrtappy.com). It’s a very solid metal handle for mobile and tablet testing that holds and elevates a webcam above the device (you can use software like Morae to “flip” the image if you need to). It’s pricey (at $295) but it’s good.