Replicability Crisis in Science?

Author

Filippo Gambarota

Replicability Crisis in Science? is one of the Specialist Courses organized from the Department of Statistical Sciences (University of Padova). The course has been proposed for the AY 2023/24 to the PhD students of the XXXIX cycle.

Overview

When evaluating the reliability of scientific findings and predictions, a major concern is about their replicability, that is their consistency across different studies aimed at answering the same question. A “replicability crisis” has been claimed in the last decades, denouncing that a large part of published research findings, especially in applied sciences, such as psychology or medicine failed to be confirmed by subsequent studies. Difficulties in moving from empirical evidence and data analysis to a scientific result push towards the need to clarify various aspects, such as rigorous definitions, the possible tools for understanding and communicating the uncertainty inherent in most scientific conclusions, the definition of non-questionable research practices. The main objective of the course is to provide a broad and interdisciplinary view, as well as the tools that may enable individual participants to focus on specific aspects of replicability that are relevant to their own discipline of interest. The epistemological, philosophical and scientific/statistical bases of replicability and of its so-called crisis will be addressed in lectures by the teachers. Guided group activities will then be proposed to the students, to discuss basic questions and apply the ideas emerged during the course to some real data analysis.

Lecturers

Schedule

Monday 8th July 2024

9:00 - 12:00

Branden Fitelson - What is a Replication?

Giovanni Parmigiani - Probability of Replication

14:30 - 16:30

Filippo Gambarota - Tools for Open Science

Tuesday 9th July 2024

9:00 - 12:00

Branden Fitelson - How not to measure replication

Giovanni Parmigiani - How to measure replication

14:30 - 16:30

Filippo Gambarota - R tools and examples for Measures of Replication

Wednesday 10th July 2024

9:00 - 12:00

Branden Fitelson - Replicability in Psychology

Giovanni Parmigiani - Replicability in Cancer Science

14:30 - 16:30

Filippo Gambarota - Meta-analysis and multiverse analysis

Groups

During each day there will be some group activites. We created some balanced groups according to the background.

Group Bayes

  • Chuchu Jia
  • Davide Forcina
  • Evgenii Pashnin
  • Jaurel Kagho zanguim
  • Maria Francesca Patalano

Group Bonferroni

  • Alessandro Zito
  • Alex Cecchetto
  • Allegra Sartore
  • Enrico Carraro
  • Yunfeng Sun

Group Gosset

  • Ambra Perugini
  • Giovanni Duca
  • Hillary Muhanguzi
  • Leonardo Genesin

Group Neyman

  • Elena Baldisseri
  • Isabella Valbusa
  • Kenenisa Tadesse Dame
  • Ludovica Natali
  • Vanshika Keshwani

Group Pearson

  • Irene Di Pietro
  • Matteo Schiavone
  • Runpeng Miao
  • Sara Costa
  • Virginia Murru

Group Tukey

  • Agmas Sisay Abera
  • Alberto Petrin
  • Denise Feurer
  • Laura Gorla
  • Matteo Licitra

References

Errington, T. M., Mathur, M., Soderberg, C. K., Denis, A., Perfito, N., Iorns, E., & Nosek, B. A. (2021). Investigating the replicability of preclinical cancer biology. eLife, 10. https://doi.org/10.7554/eLife.71601
Fletcher, S. C. (2021). How (not) to measure replication. European Journal for Philosophy of Science, 11, 57. https://doi.org/10.1007/s13194-021-00377-2
Machery, E. (2020). What is a replication? Philosophy of Science, 87, 545–567. https://www.cambridge.org/core/journals/philosophy-of-science/article/what-is-a-replication/F41B751ECC31462CBBD46711BC733AEE
Mathur, M. B., & VanderWeele, T. J. (2020). New statistical metrics for multisite replication projects. Journal of the Royal Statistical Society. Series A, (Statistics in Society), 183, 1145–1166. https://doi.org/10.1111/rssa.12572
Miller, J. (2009). What is the probability of replicating a statistically significant effect? Psychonomic Bulletin & Review, 16, 617–640. https://doi.org/10.3758/PBR.16.4.617
Nosek, B. A., Hardwicke, T. E., Moshontz, H., Allard, A., Corker, K. S., Dreber, A., Fidler, F., Hilgard, J., Kline Struhl, M., Nuijten, M. B., Rohrer, J. M., Romero, F., Scheel, A. M., Scherer, L. D., Schönbrodt, F. D., & Vazire, S. (2022). Replicability, robustness, and reproducibility in psychological science. Annual Review of Psychology, 73, 719–748. https://doi.org/10.1146/annurev-psych-020821-114157