ReproducibiliTea Online: Comparing the standard Psychology literature with Registered Reports

The worldwide spread of the SARS-CoV-2 virus – and subsequent increase in people affected by COVID-19 – is causing noticeable disruptions in our regular lifestyle. However, we are doing our best to translate OSCR initiatives online, not only to continue our mission to increase awareness on open science practices but also provide valuable support and human connection during this period of (self-)isolation.

One of our first online events will be the sixth OSCR ReproducibiliTea journal club, which will take place on April 8th at 11:00¹. We will have a special guest: Anne Scheel, Doctoral Candidate at the Department of Industrial Engineering & Innovation Sciences, Human Technology Interaction, Eindhoven University of Technology. Anne studied psychology at Heidelberg University and psychological research methods at the University of Glasgow, and worked in a developmental psychology lab at Ludwig Maximilian University of Munich for two years. Her background is in infant research but, since she first learned about the “replication crisis” in psychology, she devoted more and more time to follow the discussions around ways to make research more transparent and reproducible. Eventually this led her to switch tracks and turn to meta-science as main research focus: in October 2017, she started her PhD in Daniël Lakens’ project Increasing the reliability and efficiency of psychological science in Eindhoven.

Anne will guide us through her latest preprint²:

Scheel, A. M., Schijen, M., & Lakens, D. (2020, February 5). An excess of positive results: Comparing the standard Psychology literature with Registered Reports. https://doi.org/10.31234/osf.io/p6e9c

Registered Reports³ are designed to prevent a host of biases from distorting the evidence that makes it into the published literature. The format makes it much harder for authors to use questionable research practices such as p-hacking or HARKing, and it makes it much harder for reviewers and editors to block negative results from publication. Because Registered Reports appear to be such a powerful tool against bias, and since the format has gained a lot of popularity in a short time, a straightforward question to ask is how the results reported in published Registered Reports compare to those in the standard literature. This paper investigates the question by replicating a method by Daniele Fanelli – who reported that 91.5% of psychology papers find support for their first hypothesis – on a random sample of “standard” hypothesis-testing articles and all available Registered Reports in psychology. The results are 96% positive results in standard psychology articles compared to only 44% in Registered Reports (for similar results, see this Perspective paper in PLOS Biology). The paper discusses several limitations of this retrospective observational approach, in particular that important a priori differences might exist between the standard articles and the Registered Reports in the sample. Anne and her co-authors further present a qualitative analysis of the language used to describe hypotheses in Registered Reports, showing a surprising amount of variation and some notable differences between replication studies and original work which might be of interest to other meta-researchers.

An invitation via Outlook calendar has been sent to researchers in the OSCR ReproducibiliTea mailing list. This email includes a link to join the meeting remotely using Zoom, a popular online conferencing service. Simply click on the link and you will join the call.

During the Zoom meeting, please follow these guidelines:

• wear headphones
• mute your microphone
• video is optional (in case of connection issues, you may be asked to turn it off)
• pay attention to the moderator (which will be Antonio)
• if you have questions
  • click on the Raise Hand button and the moderator will unmute you; or
  • write down your question in the chat and the moderator will read it
• avoid talking over each other and make sure that everyone can have their opportunity to speak
• arrive 5 minutes before the beginning of the call, to familiarize with the online environment and solve possible technical issues

If you are not part of the ReproducibiliTea mailing list but would like to join the call, please contact Antonio.