Remote Usability Testing: Pros and Cons
I love usability testing but I have to admit, in-lab tests can be challenging to set up. Research facilities, a qualified moderator, participants willing to commute at a specific date and time and, let’s face it, a substantial budget are required (even though there are tricks to do user research on a budget). Today, several online platforms allow remote usability testing. Their offer is very appealing. You only need to build your test, email it to your participants and collect the automated analysis results. Is it too good to be true?
To help you make up your mind, I will play the devil’s advocate and give you five reasons to choose remote usability testing and five reasons to avoid it…
5 Reasons to Choose Remote Usability Testing
It’s cheaper than in-lab usability testing. Typically, you pay by project, task or participant. Many platforms are flexible and let you pay only for what you need. Others offer subscriptions, which is great if you do remote testing regularly. Tests are conducted online so there is no moderator, real-time note taking or research facilities to pay for.
It saves you time. There is no need to schedule 60 minutes for each participant. Tests can all take place at the same time and not necessarily between 8am and 6pm on weekdays. Also, most remote usability testing platforms analyze results automatically to provide success and failure rates for each task (e.g., find a store).
Test results are analyzed automatically. This saves time but also reduces the analyst workload. With in-lab usability testing, an analyst usually takes detailed notes in real-time or spends hours watching test recordings. He needs to calculate success and failure rates, analyze different navigation paths, find out where participants struggled, etc.
It’s more flexible. All tests can be conducted at the same time, anywhere in the world and at any time during the day or night. Participants are free to do the test wherever and whenever they want. This means you could send a test invitation to users in different countries Friday afternoon and have fifty completed tests by Monday morning.
It allows you to test with more participants. Think about it. Tests can take place simultaneously, no human support is required and you don’t need to analyze results. Instead of limiting yourself to ten participants, as it is often the case with in-lab usability testing, you can easily recruit thirty users. You could even recruit a few hundreds, which would not be possible in terms of time, effort and cost with in-lab usability testing.
5 Reasons to Avoid Remote Usability Testing
It provides limited or no qualitative feedback. When we do in-lab usability testing, we ask participants to think out loud to get spontaneous reactions. Even if you ask users to think out loud while doing your test remotely, they probably won’t do it. Therefore, qualitative feedback is limited to a few survey questions between tasks.
It’s hard to make sure participants take your test seriously. Because there is no moderator to assist participants during remote usability tests, it is up to them to decide when they are done with a task. At imarklab, we use a platform where users click on “I succeeded” or “I give up” after each task. Afterwards, the platform verifies if users really succeeded by checking if they reached a success URL, saw a specific text, etc. Some participants might pass through tasks quickly without really doing them to get the incentive (e.g., amount of money). This type of behavior distorts the data.
It doesn’t work with complex tasks and prototypes. During in-lab usability tests, the moderator can assist participants in case of difficulties. He can also help with complex tasks. This means success “with assistance” and failure “with assistance” are possible. There is no such grey area with remote usability testing. The moderator also manages expectations with prototypes, especially for graphic design. This way, he makes sure participants comment only what can be evaluated on incomplete prototypes.
Several participants will have technical issues and give up. After clicking on your test link, users have to go through a few steps before they can start doing tasks on your website. In most cases, they have to allow the platform to access their webcam and microphone, upload a plug-in and use a compatible browser. Participants who are not used to remote usability testing might have technical issues and not even start the test.
Results will not be 100% accurate. First, because of participants who go though tasks quickly just to get the incentive. Second, because some successes will be considered as failures. Let me explain. You could ask your participants to find a specific page and set up the URL of that page as the “success URL”. A participant could find the button to access that page easily and click on “I succeeded”, which sounds fine. However, it will be considered as a failure if he does not click on the button to get to the success URL.
***
Remote usability testing can be a smart choice or something to avoid. It depends on your context. Taking each test methodology strengths and weaknesses in consideration will help you choose the best one for your needs. Would you like to benefit from remote usability testing strengths while minimizing its weaknesses? We can help. Contact us to learn more about our remote usability testing service.
Are you having a hard time choosing a usability research approach? We built this amazing decision tree to help you choose the best approach!
