ReproducibiliTea Online: v-hacking

The eighth OSCR ReproducibiliTea journal club will take place on June 10th at 11:00. The discussants will be Dr. Ian Hussey and Prof. Sean Hughes, from the Department of Experimental-Clinical and Health Psychology at Ghent University (Belgium).

Ian and Sean will guide us through one of their latest papers¹:

Hussey, I., & Hughes, S. (2020). Hidden Invalidity Among 15 Commonly Used Measures in Social and Personality Psychology. Advances in Methods and Practices in Psychological Science. https://doi.org/10.1177/2515245919882903

The replication crisis has rocked many areas of scientific research: important findings cannot be reproduced. In this paper, Ian and Sean propose that psychological science may face not only a crisis of replicability but also of measurement, which may also be obscuring incorrect conclusions. Ian and Sean introduce the concept of v-hacking (validity hacking) as analogous to the established issue of p-hacking.

Accurate measurement is a cornerstone of the scientific method. In the social sciences, researchers often aim to capture phenomena they are interested in using proxy measures. For example, a self-report questionnaire that asks about specific symptoms (e.g., feelings of sadness, poor sleep, etc.), in order to capture the latent construct that is of primary interest (e.g., depression). Unfortunately, the vast majority of researchers merely assume that their scales are valid measures rather than assessing this directly in their own data. Where such assessments of validity are reported, this is most often cursory and omits the most important tests.
Ian and Sean discuss the results of a very large scale study (N = 144,000, 26 scales) that provides comprehensive assessments of many commonly used scales in Social and Personality Psychology. They provide an overview of the process of assessing structural validity – no prior knowledge of psychometric methods is assumed. Results from this study are used to answer the question of whether the observed under-reporting of structural validity results in the literature likely obscures an even larger problem: that many of our most commonly used measures demonstrate hidden invalidity. Implications for our confidence in our collective research findings are discussed, and ways to improve this situation in future research are outlined.

Despite the fact that the paper focuses on specific measures in a specific field of psychology, other areas of psychology and even other disciplines can be inspired by this work and examine their own instruments and measures.

An invitation via Outlook calendar has been sent to researchers in the OSCR mailing list. This email includes a link to join the meeting remotely using Zoom, a popular online conferencing service. Click on the link, insert the password provided in the invitation mail, and you will join the call.

During the Zoom meeting, please follow these guidelines:

• wear headphones
• mute your microphone
• video is optional (in case of connection issues, you may be asked to turn it off)
• pay attention to the moderator (which will be Antonio)
• if you have questions
  • click on the Raise Hand button and the moderator will unmute you; or
  • write down your question in the chat and the moderator will read it
• avoid talking over each other and make sure that everyone can have their opportunity to speak
• arrive 5 minutes before the beginning of the call, to familiarize with the online environment and solve possible technical issues

If you are not part of the OSCR mailing list but would like to join the call, please contact Antonio.

Please remember that attendees of every OSCR event (in-person or online) are required to follow our Code of Conduct.