ReproducibiliTea Online: The Generalizability Crisis

The fourteenth OSCR ReproducibiliTea journal club will take place on March 12th at 15:00 CET. Our special guest will be Luisa Fassi. After obtaining her Bachelor degree in Psychology at Erasmus University Rotterdam, Luisa graduated in Psychological Research at the University of Oxford, where she recently received the Humphrey Prize for Best Research Dissertation. Luisa is a junior representative of the Rotterdam R.I.O.T. Science Club for Reproducible, Interpretable, Open and Transparent Science, and is an active contributor of the ReproducibiliTea initiative at Oxford.

Luisa will guide us through a recent influential paper on the issue of generalizability¹:

Yarkoni T. (2020). The generalizability crisis. Behav Brain Sci. 2020. https://doi.org/10.1017/S0140525X20001685 [Preprint on https://doi.org/10.31234/osf.io/jqw35]

In this article, Dr. Yarkoni highlights a pervasive issue within psychology that also occurs across different biological and medical sciences. Shortly put, the issue is the following. Suppose that a researcher formulates a theoretical hypothesis (e.g., “physical exercise increases sustained attention”). To empirically test such hypothesis, she runs an experiment on healthy participants. She operationalizes sustained attention as “response times to a specific set of stimuli (e.g., arrows) in a 30-minute task consisting of four blocks”. Once results are collected, statistical analysis is run to examine whether the experimental manipulation influenced the investigated outcome. Based on the significant effect size, the researcher concludes that physical exercise increases sustained attention. However, a fundamental assumption underlies this finding’s discussion. Namely, the same result would be expected to emerge if the study was run using an experimental task with circles instead of arrows as stimuli or five instead of four blocks. Dr. Yarkoni argues that there is often a mismatch between researchers’ generalization intentions and the statistical operationalization of their theoretical hypotheses. Experimenters usually model variability by including “random effects” for subjects in their statistical analysis. In this way, they can generalize the observed effects beyond the tested sample. However, only a few researchers accord the same treatment to other variables they clearly intend to generalize over (e.g., stimuli, tasks, or research sites). For instance, in the example above, no generalized statement regarding sustained attention as a general construct shall be made unless the statistical procedure explicitly models random effects for the task and stimuli employed. Arguably, this critical issue stands at the heart of both the generalizability and reproducibility crisis. After discussing its multifold implications, we will discuss practical and theoretical considerations for overcoming it.

An invitation via Outlook has been sent to researchers in the OSCR mailing list². This email includes a link to join the meeting remotely using Zoom. Click on the link, insert the password provided in the invitation mail, and you will join the call.

During the Zoom meeting, please follow these guidelines:

• wear headphones
• mute your microphone
• video is optional (in case of connection issues, you may be asked to turn it off)
• pay attention to the moderator (which will be Antonio)
• if you have questions
  • click on the Raise Hand button and the moderator will unmute you; or
  • write down your question in the chat and the moderator will read it
• avoid talking over each other and make sure that everyone can have their opportunity to speak
• arrive a few minutes before the beginning of the call, to familiarize with the online environment and solve possible technical issues

Please remember that attendees of every OSCR event (in-person or online) are required to follow our Code of Conduct.

Take care,

Luisa Fassi and Antonio Schettino