Experiment Design
During the design phase of an online experiment you always need to consider the differences to a lab environment. There are slow contact possibilities in case of a problem, higher dropout rates and unpredictable influences for a participant. Most of the best practices in terms of participant grouping for large scale online experiments can be used in lab experiments as well.
Participant Grouping
Match participants as late as possible - Participants on MTurk for example tend to drop out early in the experiment, if they do. For that reason you should ask participants to complete any individual tasks in your experiment prior to the interactive actions, as long as it does not interfere with your experiment design.
Use dynamic grouping - To accomplish a late grouping, you need to wait for participants to reach a certain stage of your experiment. At that point participants should wait for the required number of participants to proceed. In some cases it might take a while to reach the number of required group members (e.g. large groups).
Specify waiting information - As the participants do not know, if other participants will join the experiment or already did, you need to provide clear instructions. Provide a fixed waiting time to prevent participants from waiting hours and send them to a timeout step afterwards. That will prevent complaints and keeps participants more willing to wait. You should also display the amount of participants that are already waiting to get grouped for similar reasons.
Most of the requirements are covered by using the SoPHIELabs Waiting Room. If that does not cover all your use cases, you can create individual groupings as well.
Participant Attention
Improvements
As there is little knowledge about the distraction of a participant, it is necessary to inform participants about every change. Participants will be on the desktop or even away from the computer, while they get grouped. If you are doing time-sensitive research, where participants need be on spot right after grouping you can use acoustic signals to get the attention back to the experiment. Also use an appropriate countdown to allow for a visual signal, which is more likely to be noticed than a static message.
Information
Whether in the classroom or in an online experiment, information about the participants is crucial at any point. For that SoPHIE provides information about the participant status. Whenever a participant loses the connection, closes the browser or is inactive for a certain amount of time, the participant symbol in the Session Administration turns red. That also includes information about the time passed since last contact.
If a participant switches to a different browser tab or minimizes the browser, the participant symbol immidiately turns yellow. As this information is also available within the experiment, you can exclude participants from grouping based on that information.
Participant Requirements
Depending on your experiment design, participants may be required to match certain criteria. For time-sensitive tasks you might want to exclude participants with a bad connection to the experiment server or you need to check for camera availability, when participants are asked to interact through video chat. You can also prevent participants from participating with mobile devices or allow / disallow certain browsers.
All this can be restricted by using the SoPHIELabs Requirementscheck Steptype.
Payoff
The specific preparations necessary to set up the payment for Amazon MTurk are already discussed in this article. Please consider that participants that are looking at the list of HITs can only see the fixed reward they will get. As research experiments tend to have a high payment in comparison to the platform average that is not a big concern, but you want to keep that in mind for smaller tasks.