Online Data Collection

Recommended experimental protocol:

• Please ensure your IRB protocol covers your experimental procedures
• Use Qualtrics to code the survey/experiment and then post that link to Amazon Mechanical Turk (MTurk, for a general how-to on MTurk: see here and here)
• For a great how-to on Qualtrics and programming a psychological experiment: see here
• For tips on coding in HTML: see this document
• In the Human Neuroscience Laboratory, we have direct experience coding up experiments employing both word and picture stimuli testing old-new recognition memory, recall, source memory (e.g., for spatial location), as well as a variety of creativity tasks (e.g., Alternate Uses Task and Remote Associates Task). For templates contact Preston. We also have experience with counterbalancing and looping in Qualtrics (e.g., see this video for a short how-to on looping), which is necessary for appropriate and efficient experimental designs!

• If using Qualtrics, there are FOUR things that are critical:
  • 1a) Ensuring that you monitor off-task performance (for related publication, Permut et al., 2019)
    • a) See here for the code to add in your Header (see the 'Look and Feel' and then 'General' section of your survey)
    • b) See here for the code to add in your Footer (see the 'Look and Feel' and then 'General' section of your survey)
    • c) Make sure to add the Embedded Data fields as detailed in Permut et al. (2019)
  • 1b) The code listed in 1a, in certain cases may not work, due to compatibility issues in Qualtrics
    • See here for another option (backup)
  • 2) At the end of online experiments, each MTurk worker is provided a unique code which they are asked to input into the MTurk survey. They must do this before submitting the Human Intelligence Task (HIT) to receive payment. If participants do not enter this code, payment is not given.
    • For example, build a final question that looks like this
  • 3) To ensure you create a Qualification in MTurk to ensure MTurkers cannot retake your experiment
    • Why do you need this? See here and here
    • To get your unique MTurk code: http://uniqueturker.myleott.com/
    • There are additional qualifications/restrictions you can setup in your MTurk requester account when posting your study, see here for the Human Neuroscience Laboratory settings (as you can see in the picture, I have set a restriction such that the study has not been participated in before, here called, 'Remembering Lists')
      • E.g., We like recruiting participants with a HIT approval rate of >95% (meaning that those who participate are those that are likely to finish it as that is their prior history), and also having participants with at least 50 HITs approved meaning they have done a considerable amount of MTurk studies (note that these setting follow other published studies).
      • Note also that some restrictions, like age, cost additional money (so called, 'Premium Restrictions')
  • 4) How to save/extract your data (e.g., as a .csv file)
    • Turn on variable naming using the gear button next to the question. Check box called 'variable naming': this will save the actual response selected as opposed to the position of the click or choice.
    • In the export section make sure to export the .csv with the 'choice text' selected (see also additional options like removing blanks)
    • Point is, do not assume all your data/responses will be saved. Ensure the output captures it!

Alternatives to Qualtrics

• You can code in Java/HTML straight in MTurk, however that is not GUI-based and can be time consuming to learn and do
• If interested in this approach, see these very helpful videos and tutorial
• In addition, you can also use REDCap or PsyToolkit
  • REDCap is more survey-based. Research Services @BC are available to meet with members of the Boston College community to discuss individual REDCap projects. Individual consultations or customized class consultations are available by emailing researchservices@yj1001.net or dalgin@yj1001.net.
  • PsyToolkit has an amazing library of classic cognitive psychology experiment templates some of which you think would not be suitable for online data collection (e.g., Stroop Task, Mental Rotation, Posner Cueing Task, Wisconsin Cart Sorting Task, Go/NoGo task, Inhibition of Return, Multitasking, etc.)

(Not Comprehensive) Checklist for Online Data Collection

These are suggestions and not rules (see here for additional information)

1) Hiding sensitive content in your client-side code

• Check your client-side code for sensitive content. Open your experimental procedure in a browser window. If you are using Chrome, click View > Developer > View Source. In Firefox, click Tools > Web Developer > Page Source. (Google “how to view page source in [browser name]” to find instructions for other browsers.) A new browser window will open with the source code for the particular page of your experiment you are viewing. Scroll through this code, and make sure there isn’t anything in there you wouldn’t want participants to see (e.g. hidden payoff values, logic that reveals your experimental manipulation, names of different conditions, etc.).
• Obfuscate any JavaScript code that you are sending client-side so participants can’t view hidden values or the logic of your procedure by viewing the source code. Open-source tools like javascript-obfuscator created by Timofey Kachalov can be used to obfuscate your JavaScript code. A Web UI for javascript-obfuscator@0.18.6 can be found at obfuscator.io. (A better option is to keep all sensitive content server-side, but this isn’t always possible.)

2) Testing your procedure

• Create a Mechanical Turk worker account so you can test your procedure as a participant before launching publicly
• Set a custom qualification and assign it only to your own worker account
• Launch your experiment with the custom qualification assigned to your worker account, preventing other workers from viewing the HIT
• Log into your worker account, and complete your HIT at least 8 times, each time using one of the following 8 configurations (Browser + Operating System):
  • Chrome + Mac
  • Chrome + Windows
  • Firefox + Mac
  • Firefox + Windows
  • Edge + Mac
  • Edge + Windows
  • Safari + Mac
  • Safari + Windows
• Either block mobile devices, or test your HIT on mobile to ensure your procedure functions as intended. (You can simulate a mobile request in Chrome by selecting View > Developer Tools, clicking on the Toggle Device Toolbar icon, selecting “mid-tier mobile” and selecting a mobile device model in the device toolbar that pops up. More information can be found on this page of the Google Developers Help Center.)
• Check the URLs for each page in your procedure to ensure they do not contain any information that might reveal your experimental manipulation
• Check your logged data to make sure all activity is being tracked appropriately
• Have a few friends complete your survey and time the duration of their responses to each question type, so you have some sense of how long it *should* take the average participant to complete the procedure

3) Protecting yourself against low quality data

• Always include a neutral answer (e.g. “Choose here”) as the preselected or default response in form elements such as drop down lists
• Test multiple response styles (e.g. radial, slider, drop down) to determine whether response type may bias results
• Do not advertise eligibility requirements as part of the study sign up process, instead use qualification restrictions (such as country, number of studies completed, etc.) or a pre-survey to restrict eligibility (Siegel et al., 2015)
• Prevent retakes by assigning custom qualifications through the Mechanical Turk API, or using Qualtrics’s built-in system
• Collect IP addresses and location data, as these can be used to detect bot activity
• Use a transcription task at the start of your experiment that requires participants to transcribe a photograph of handwritten words
• Use a comprehension check with a non-obvious pattern (i.e. answers to multiple choice comprehension questions should not all be assigned to the top radial in each choice list)
• Include a mandatory free-form response at the end of your survey (you can instruct participants simply to type “na” if they have no response/comments)
• Use custom JavaScript to count the number of times a subject leaves the page, to keep track of how long the subject spends away from the page, or even to display a pop up message when the participant leaves the page, warning her that the HIT will not be approved if she leaves the procedure before completing it. (You can find some sample code that implements this sort of tracking in Qualtrics, see above)

4) Reducing the negative impact of attrition

• Ask personal demographic questions, and announce chances for additional financial incentives at the start of your procedure, as both of these measures have been shown to reduce dropouts (Reips, 2002a)
• Introduce a high hurdle task by concentrating motivationally adverse factors toward the start of the procedure
• Ask participants to estimate the likelihood they will complete the whole experiment before they begin the procedure
• Use a prewarning with an appeal to conscience (e.g. “Our research depends on good quality data, and that quality is compromised whenever participants fail to complete the full procedure.”) (Zhou & Fishbach, 2016)
• Use practice trials, pilot new experimental protocols, or ask participants to complete a boring task prior to starting the focal experimental procedure

5) Analyzing your data

• Wait until all data have posted to your server, or deactivate your Qualtrics survey, before beginning your analysis – some platforms (e.g. Qualtrics) do not post unfinished responses to the server for up to one week after a participant initiates her response, and you will not be able to measure attrition rates until these data have been posted
• Check every free-form response you have collected, looking for suspicious patterns (e.g. one script I’ve detected in my data copies text from the surrounding page, mixes up the words, then pastes the result into the free-form response box)
• Check for repeated IP addresses and latitude/longitude coordinates, as significant numbers of such repeated information may be a sign of bot activity (Bai, 2018)
• Include browser and operating system as factors in your analysis, as system configurations may impact participants’ interaction with your procedure, and have been found to be correlated with important individual differences that may affect your results (Buchanan & Reips, 2001)