Recommendations and Best Practices in Mobile Usability Testing

NOTE: Much of this content is based on two earlier blog entries I wrote here a while back in 2009, so I added some additional and more recent details to come up with this revised version…

Prescreening users

First, prescreen your test participants. You need to know that they actually are capable of using the mobile devices that they own before you test. If you don’t do this for your test subjects, you may get a participant who has no clue as to how they can use their device beyond the basic function they do every day. If you don’t prequalify, you will get some “noise” in your test data that may make the whole session with that person worthless.

Recruit people who use the device, not just people who “own one”

Sometimes you will encounter a participant who got through the screening process and has NO IDEA how to do anything with his mobile device other than make or receive calls. You won’t get a lot of valuable data from this person, because a good half of your time with them will be spent explaining how to do things with his phone. Make sure your screener has questions to prevent these type of recruits from showing up.

Understand usage and context

In the pretest conversation, get a sense of how people actually use their mobile devices. Asking them what they do with their phones allows you to get a better sense of the priorities and features the user needs and provides valuable context that can inform your design goals.

Capture Everything

I’m a big fan of Morae from TechSmith, an app designed to record and analyze usability tests, and the latest version supports two video sources. The setup is simple: Use two cameras, one that is on a mounted tripod pointed down at the device, and the second monitoring the participant. Morae captures both video signals and records them, as well as transmitting the video of the session to a note taker via your work network.

Get comfortable with the devices you are testing

Even if you are recruiting users that are familiar with and/or are “power users” of a particular type of phone, you need to spend time getting used to how the device works. You may encounter a person who is not accustomed to the eccentricities of how the phone operates in the space you are testing (for example, a downloadable app on a blackberry). Be ready to work through such bumps by knowing how to do

When possible, test no more than two devices in one round of testing

If you are going from one device to another, you tend to lose focus on what you should be doing which is observing how the participants uses the app you are testing – you will focus more on how the device works than you should. So, test two device types in each usability test round, and try and have all the participants of Device A test on Day one and all of Device B test on Day 2.

Remember to separate the device’s usability issues from the application

On some phones you are going to see some very painful interactions that the participant has to go through to accomplish the task you have laid out for him or her. More often than not that is caused by the awkward data entry mechanism that the phone has and NOT the app itself. Always remove the usability problems the device causes from issues observed with what you are testing. If you’d like to catalog those issues and send them to LG or Nokia, fine, but that’s NOT your priority.

Make sure what you are testing supports the phone’s native controls

A couple of show-stoppers came up in a previous test when the participant used the native buttons/controls while accessing the app (the keypad’s “Back” button was one example). The app was not able to support this and logged the users out. If you have at least one participant do this, odds are that many more will being doing this as well. Make sure to note this as a MAJOR issue that the dev team needs to fix.

Creating prototypes to test

You may be testing a new design instead of an existing mobile app or site. You can use any design tool to create a clickable prototype, and can create either an html “mini-site” or a clickable PDF. Here’s some links (there’s a lot more out there):

Omnigraffle: http://viget.com/inspire/how-to-create-prototypes-with-omnigraffle

Axure: http://axureland.com/axure_blog/entry/mobile_prototyping_with_axure_rp_6.0

Keynote: http://blog.amirkhella.com/2010/07/13/teaser-iphone-running-an-interactive-prototype-built-with-keynote/

Getting the prototypes on the phone

You can use PDF readers to open PDF prototypes. For iPhone, I’ve used Keynotopia and for Android I’ve used a couple of different apps (whose name escapes me right now). Some PDF readers for Android don’t “pick up” the jump to links in the PDF, so be sure to test the prototype on the device before bringing in users. You can either copy the files to the phone or if that is not possible you can access the files through Dropbox on the device.

Designing on the device itself

A couple of apps have come out over the past few years that allow people to design or test paper sketches on the device itself. One iPad app called Blueprint allows you to design iPad and iPhone apps and export the results to a special reader/viewer. Another app called POP allows you to photograph sketches and then embed links that users can use to jump to different pages and views.

Holding the device

I’ve used – and like – Mr. Tappy (http://www.mrtappy.com). It’s a very solid metal handle for mobile and tablet testing that holds and elevates a webcam above the device (you can use software like Morae to “flip” the image if you need to). It’s pricey (at $295) but it’s good.

What is the next frontier in mobile UX/UI design?

Three things that are going to be a big focus for mobile UX/UI designers in the years to come:

Voice Interaction: Accessing information from your mobile device or the cloud through speech is here now, but we "turn it on" to do so. I expect this to change in the future, and that your mobile phone to always be "on call" and "wake up" when spoken too. I also expect the information and functions that will be accessed through voice commands to become more complex, useful and proactive: think "digital butler." Not a lot of "UI design" here, but a lot of UX design thought needs to go into making these experiences work seamlessly and well.

Smart Devices: See the "digital butler" note above, and think "service design" for end-users… smart devices that learn suggest and help (sending a text to someone if you are running late for a meeting, for example.) Again, more process and AI workflows than UI.

Extending and/or tethering: Think "wearable computers", with devices like the Pebble wristwatch that connects to your mobile phone via low-powered bluetooth, allowing the user to make calls or see messages on his or her wrist. Also think about how mobile devices can engage with devices or locations in your home (opening a grocery list app when you stand next to your pantry for a long while, or "talking" to your smart TV while you are watching a show). Lots of potential UIs that need designing here…

See question on Quora

What are the big topics in mobile user interface design discussions in 2013?

Web vs. native apps. Still. While the majority of companies are pretty firmly in the "native app" space, there are still debates and discussions taking place around HTML 5 and it's capabilities. The problem is, HTML 5 can't do everything a native app can, and it also (if not tweaked and optimized) doesn't provide the same type of fluid experience that users expect. I've seen many attempts at combining the two by using a thin "wrapper" around HTML5 content (most famously from LinkedIn and Facebook), but  the "killer app" using such an approach has still not come out. People will still try, though.

Responsive Web Design. I heart Responsive Web Design. My personal blog is responsive, and I can see many many reasons for companies to make sure their app is responsive. However, the sad fact remains: existing companies already HAVE web sites, and changing these sites to make them responsive is a big investment of time and money, and the benefits aren't ones that align very well with most "20th Century" business drivers. It's going to be an uphill climb (I've fought for RWD on projects, and lost, because "it wouldn't have any noticeable ROI or impact on customer satisfaction").

What platform to design for? Do you design for Android and iPhone, or focus on only one? To again belabor the obvious, companies have limited budgets, and they look at where the users are. Right now, there's a LOT of Android users out there, but if you dig deeper into the stats you find that they aren't buying apps and don't actually use the device nearly as much as iPhone users do. And what about Windows Phone? What apps to create and for what devices is a big topic of discussion at companies big and small around the world.

See question on Quora

Lessons in #UX: It’s not about “channels”, it’s about doing things

I’ve been in lots of business meetings discussing mobile and/or tablet “strategies” lately, and I often shake my head at what I hear proposed by people who, frankly, should know better. Too many of these proposals are about features and channels and devices, and that’s a great way of think about things… if it was 1994 and the #1 tech company was Microsoft. As I glance over at my desk calendar, I note that it’s now 2012 and the #1 tech company is currently Apple.

Times have changed.

I love features, and as a technologist I also love the latest and greatest gadgets. But when I hear people talking about “channels” and how they should focus on one particular channel to gain market share I sometimes reply, forcefully, “It’s not about channels, it’s about doing things.” Users arent technology geeks, or product managers… they just want to do things quickly and simply. They don’t think like companies or (way too many) product designers do.

Two examples, and both are “cross channel” experiences. Example one: I, like many people, use iTunes to organize my music and media. I am a “tweaker,” and I like to update the metadata of my collection to be as accurate as possible. I even go to Wikipedia to make sure the genre that I assign a particular artist is correct (I know, but I can’t help it, I’m a tad ADD). As a tool, iTunes has been incredibly useful in this effort and I have been a (mostly) happy user of the app.

Then I started using iTunes Match, the cloud-based service that helps me sync my music collection across devices.

And it royally screwed things up.

I would make a change in a listing (genre, artist, etc.) and then I’d come back later and see my change went away and things reverted back to the way it was before. How? Because the older data in the “cloud” was apparently more important than my newer revision. Here you have a “cross-channel” experience that as a user experience professional I understood and, when it worked, appreciated. But as a user, it was (and is) a frustrating experience.

The lesson? I think the iTunes Match team, like many product managers I work with, have focused on the wrong thing – the cloud, not the end-user experience. Most users don’t care how things like this work, they just want it to not screw up and “break” stuff they have and use. If I change something, I want it to stay changed. Period.

Example number two: Dropbox has a very simple offering, a syncing service that (like iTunes Match) keeps certain files the user selects synced across multiple computers. That’s it. It’s simple, and just works. It has become a vital part of my workflow, and supports my day-to-day work and personal activities because it doesn’t overcomplicate things. It does one thing very well and (here’s the important part) gets out-of-the-way. It works across channel, across device, and does a great job.

Final thoughts: When it comes to designing solutions for users, the challenge and the opportunity is to align with what they need to do and give them tools to help them do it. I have often told my peers that this is the hard part of user experience design, and that creating UI screens is easy. I say that because you have to really focus on making an offering that “fits” with how people think and do things, and the only way that “channels” matter is the context of use… WHEN and WHERE the user accesses your solution.

Any additional emphasis on “channels” should be limited to business presentations and financial planning.

Native Mobile Apps Offer Advantages Over “Wrapper Apps” for Financial Services

UX

Originally published on Fiserv’s “The Point” Blog, July 10th, 2012.

It isn’t much of an exaggeration to say that mobile apps have changed everything. The ways people work, shop and live have been enhanced by dedicated applications that support their specific needs. The advertising tagline from Apple – “there’s an app for that” – has become less about marketing and more of an accepted truth, reflecting the wide-ranging impact of mobile apps on our lives and work.

Not all apps are created equal, however. Many apps are poorly designed and are quickly deleted by users after one try. Some apps are elegantly designed but lack a solid infrastructure to support the user. And others, well, they aren’t really apps at all. They are websites, packaged as apps. These “wrapper” apps are shells that are downloaded and installed on the device. The shell then accesses mobile formatted web content from a traditional web server.

In some instances, such as a news or sports app primarily used to display online content, wrapper apps are perfectly fine. For others, such as a mobile banking app intended to enable more interaction, not so much. When you compare a native app designed for a specific mobile platform to a wrapper app, native apps offer some significant advantages. The primary one is the performance of the app itself. A native app resides on the device and doesn’t have to “re-download” itself every time it’s opened. A wrapper app on the other hand does have to download the Web forms and screens that aren’t preloaded in the app itself. This means a native app is faster and highly available, which is critically important for financial services.

The second factor is interactivity. As good as mobile Web technology has become, it is not always as responsive as a native app, or as powerful in terms of the capabilities it can support. For example, wrapper apps cannot enable the deposit of a check via a camera-equipped smart phone, a capability that is increasingly in demand. Wrapper apps can’t take advantage of some of the core code that is in the operating system of the phone itself – code that allows for smooth transitions, graphing and animations that are an inherent part of a good user experience.

To be fair, there are some advantages to wrapper apps. Initial costs can be lower, as Web forms take less time to code and deploy. In addition, the same forms can be presented across multiple phones, served up through different wrappers. If you choose to develop a wrapper app, the most important factor is to create a “homogenized” user experience, in which case web technologies such as HTML and CSS can be used where appropriate along with native capability where the user experience requires.

When evaluating your mobile financial services strategy, keep in mind the experience your user expects and choose the technology that will enable your financial institution to deliver that experience.

What are best practices for mobile UX?

UX

I think you should always include and consider general best practices in user experience design when designing for mobile, as they are foundational. Know your user, test your designs, refine based on user testing… all these should be applied when working on mobile. Now, mobile specific best practices:

Keep context in mind. Whatever you are designing for a mobile device will be used "on the go" and you need to identify a "context model" that is applicable. I had a good example recently when I was debating a mobile design with a colleague. The colleague had created a very simple stepped approach at data entry which on the surface was fine… but it was designed without thinking of the context of use….when the user would be using the screens, and what their focus would be. This context pointed towards a different design approach.

Align with device conventions. If you are designing an app for an Android phone do not use iPhone like controls, and vice-versa. Users have "learned" the interaction patterns that is used on the device, and forcing a convention different than what they understand will produce confusion and difficulty where it can be avoided.

Test on the device whenever possible. Either create a clickable prototype or have one of the development team code a thin front-end. Testing interactions on a touch screen requires you to actually TOUCH the screen… any other test method will only give you a partial and hollow result. And test "in the wild" if possible.

Don't try and mirror the online experience. There are functions that, frankly, don't make sense to provide in a mobile app. On a recent mobile project we had requirements that mandated that a comprehensive administrative set of functionality should also be available on the mobile app – I'm talking about account permissions, rules around who gets to use what functionality, you name it.  We interviewed users and found that if we had designed all these screens they would be hardly be used, and that the more appropriate mobile design would be a much smaller subset of what was requested.  We COULD have spent a lot of time and energy supporting the initial requirements… but understanding usage patterns and context lead us to a more appropriate mobile solution.

Know your platform. You can't design for the iPhone if you don't know how it works… And how it doesn't. Every platform has its strengths and weaknesses, and you should know what you are working with.

Design for the "Immediacy of Now." Users often have to quickly accomplish one core function in a mobile app – make a payment, check for new messages, etc. Surface that function in an obvious way and make it simple and obvious. Design to support this user’s core need, and don’t add superfluous functionality that distracts from this core.

See question on Quora