Twitter Updates for 2009-06-16

  • Good morning Internets! #
  • Reviewing video from last week's user research, taking notes… #
  • Had a couple of good meetings, successfully pitched my next research project, and now, I'm off. I'm also leaving. #
  • Diane, it's 6:15 PM. Just read on the net that Kyle Maclachlin is working to bring back Twin Peaks. Also, 'How's Annie?' #
  • New at, Usability for Evil: Six Flags Online Meal Deals… #UX #sixflagssucks #usability4evil #

Powered by Twitter Tools.

Lessons in Mobile Testing: Before you test, know your users (and don’t artificially limit research)

Just got back from the Usability Professional Association conference in Portland, Oregon and had a great time meeting some of my UX peeps and attending some really interesting panels. One panel, though, got me thinking…

It was a session on mobile testing and research, and in it the two speakers detailed their research protocol (done before testing). They recruited several users and focused on capturing information on how users searched for information on their devices. Their data capture process was to have the study participants call and leave voice mail reports whenever they did a search (good, but could be better) and respond to an SMS survey at the end of each day (bad). I’ll explain my criticism…

The idea behind the voicemail was that it allowed the researcher to get context and emotional responses from the users… Obviously, the participants were able freely express any issues they encountered in the messages they left and a lot of rich data was gathered. However, there are some problems with this approach (and the panelists themselves referred to this).

The main issue was that the researchers had to transcribe the notes for data analysis from all the voice mails, an intensive manual effort. This also (in all probability) resulted in a lot of noise in addition to the data, requiring additional analysis time. The second part of the data gathering, however, is the part I find to be needlessly complex and confining.

Because part of the research protocol was to have participants use SMS messages to reply to a daily survey, they recruited only expert users… which meant that a huge chunk of the population who do not use/know how to send SMS messages were cut out of the research pool. I just spent a month interviewing young people, age 20-30, and several participants did not know how to use SMS. So the research approach prevented these users from being recruited and participating.

To be fair, it could also have been determined that experienced users were the target user for the app tested and thus the constraints were intentional. Except… going back to my recent research, several of the people I interviewed did not use SMS – they have always had smart phones, and they used e-mail instead. So you are not only losing “novice” users, you may also be losing “experienced” users – who are “experts,” but the way than they recruited participants, would not have been included.

I would have done this research a little different. Instead of voicemail, I would have users keep a log/diary of their daily mobile activities – ALL of them, not just search – and give them a notebook of 5×7 index cards bound at the top of the cards, one per day. The participants would enter the activity, whether it was successful or not, where they were, and any notes/opinions. Not as much data per “note”, but MORE data… and data about search in the context of other mobile applications. Context matters.

Another benefit of the data notebook idea is that, once removed from the binder, it is ready to card sort and do affinity exercises with. This data informs the creation of the test scenarios you develop to test the mobile application, making the scenarios more realistic and practical. Finally, the idea of the mobile survey, via SMS? Use paper, or have them fill it out in a Word document… don’t use technology when pen and paper will do just as well.

I could go on, but I want to reinforce that the intent of this research was exactly right, even if I think they may have limited their participant pool unnecessarily due to the research protocol. They spent time trying to understand how the users used the devices and the software before they tested it, and this is invaluable.

Of course, hindsight is 20/20 – I did not have an opportunity to discuss this with the panelists after their speech, and I’m sure that they would have some additional context that could have convinced me my criticism was off-base. But I still think my recommendations are good ones: Don’t let your research protocol create artificial constraints. The same data can be gathered – or more data, if you reduce the technology use, keep things simple, and cast a wider net.

Twitter Updates for 2009-06-14

  • Returned home from #PDX to find the water in our white trash pool was greener than my grass. Algaecide & two pool filters later, all is well #
  • @mindykaling Yup, I'm gonna go see it again this afternoon (listening to UP soundtrack now) – Cried like a little baby. in reply to mindykaling #
  • Just put the ol' stars and stripes out to celebrate Flag Day… #
  • New post- Lessons in Mobile Testing: Before you test, know your users (and don't artificially limit research) #UX #mobile #

Powered by Twitter Tools.

Twitter Updates for 2009-06-13

  • Just got my facebook username ( and heading to the #pdx airport and then home. Yes, it is 3:45 AM. #
  • @AmyCueva Heh. Yeah, my wife and kids are going to visit my mom and sisters for the next 2 days. At least my dog will be happy to see me. in reply to AmyCueva #
  • Back in the #ATL. #
  • Got home to find the Star Trek 1GB Wrist flash drives I ordered (from Kelloggs) were here. This excited me FAR more than it should have. #
  • RT: Using Comics In The Design Process Slides from the #upa2009 tutorial by @amycueva, of Mad*Pow (via @fellene) #
  • Christopher Lee has been knighted! AWESOME!!! #
  • More great geek news: Writer Peter Morgan (Frost/Nixon, The Queen, Last King of Scotland) has been signed to co-write next James Bond film. #
  • @Televixen I just usually point at you and say "yes, THAT girl" and then the theme song for the old Marlo Thomas show starts playing… in reply to Televixen #

Powered by Twitter Tools.

Twitter Updates for 2009-06-12

Powered by Twitter Tools.