Just got back from the Usability Professional Association conference in Portland, Oregon and had a great time meeting some of my UX peeps and attending some really interesting panels. One panel, though, got me thinking…
It was a session on mobile testing and research, and in it the two speakers detailed their research protocol (done before testing). They recruited several users and focused on capturing information on how users searched for information on their devices. Their data capture process was to have the study participants call and leave voice mail reports whenever they did a search (good, but could be better) and respond to an SMS survey at the end of each day (bad). I’ll explain my criticism…
The idea behind the voicemail was that it allowed the researcher to get context and emotional responses from the users… Obviously, the participants were able freely express any issues they encountered in the messages they left and a lot of rich data was gathered. However, there are some problems with this approach (and the panelists themselves referred to this).
The main issue was that the researchers had to transcribe the notes for data analysis from all the voice mails, an intensive manual effort. This also (in all probability) resulted in a lot of noise in addition to the data, requiring additional analysis time. The second part of the data gathering, however, is the part I find to be needlessly complex and confining.
Because part of the research protocol was to have participants use SMS messages to reply to a daily survey, they recruited only expert users… which meant that a huge chunk of the population who do not use/know how to send SMS messages were cut out of the research pool. I just spent a month interviewing young people, age 20-30, and several participants did not know how to use SMS. So the research approach prevented these users from being recruited and participating.
To be fair, it could also have been determined that experienced users were the target user for the app tested and thus the constraints were intentional. Except… going back to my recent research, several of the people I interviewed did not use SMS – they have always had smart phones, and they used e-mail instead. So you are not only losing “novice” users, you may also be losing “experienced” users – who are “experts,” but the way than they recruited participants, would not have been included.
I would have done this research a little different. Instead of voicemail, I would have users keep a log/diary of their daily mobile activities – ALL of them, not just search – and give them a notebook of 5×7 index cards bound at the top of the cards, one per day. The participants would enter the activity, whether it was successful or not, where they were, and any notes/opinions. Not as much data per “note”, but MORE data… and data about search in the context of other mobile applications. Context matters.
Another benefit of the data notebook idea is that, once removed from the binder, it is ready to card sort and do affinity exercises with. This data informs the creation of the test scenarios you develop to test the mobile application, making the scenarios more realistic and practical. Finally, the idea of the mobile survey, via SMS? Use paper, or have them fill it out in a Word document… don’t use technology when pen and paper will do just as well.
I could go on, but I want to reinforce that the intent of this research was exactly right, even if I think they may have limited their participant pool unnecessarily due to the research protocol. They spent time trying to understand how the users used the devices and the software before they tested it, and this is invaluable.
Of course, hindsight is 20/20 – I did not have an opportunity to discuss this with the panelists after their speech, and I’m sure that they would have some additional context that could have convinced me my criticism was off-base. But I still think my recommendations are good ones: Don’t let your research protocol create artificial constraints. The same data can be gathered – or more data, if you reduce the technology use, keep things simple, and cast a wider net.