Degrees of Freedom in Planning, Running, Analyzing, and Reporting Psychological Studies: A Checklist to Avoid p-Hacking

doi:10.3389/fpsyg.2016.01832

Review

. 2016 Nov 25:7:1832.

doi: 10.3389/fpsyg.2016.01832. eCollection 2016.

Degrees of Freedom in Planning, Running, Analyzing, and Reporting Psychological Studies: A Checklist to Avoid p-Hacking

Jelte M Wicherts¹, Coosje L S Veldkamp¹, Hilde E M Augusteijn¹, Marjan Bakker¹, Robbie C M van Aert¹, Marcel A L M van Assen¹

Affiliations

PMID: 27933012
PMCID: PMC5122713
DOI: 10.3389/fpsyg.2016.01832

Review

Degrees of Freedom in Planning, Running, Analyzing, and Reporting Psychological Studies: A Checklist to Avoid p-Hacking

Jelte M Wicherts et al. Front Psychol. 2016.

. 2016 Nov 25:7:1832.

doi: 10.3389/fpsyg.2016.01832. eCollection 2016.

Authors

Jelte M Wicherts¹, Coosje L S Veldkamp¹, Hilde E M Augusteijn¹, Marjan Bakker¹, Robbie C M van Aert¹, Marcel A L M van Assen¹

Affiliation

¹ Methodology and Statistics, Tilburg University Tilburg, Netherlands.

PMID: 27933012
PMCID: PMC5122713
DOI: 10.3389/fpsyg.2016.01832

Abstract

The designing, collecting, analyzing, and reporting of psychological studies entail many choices that are often arbitrary. The opportunistic use of these so-called researcher degrees of freedom aimed at obtaining statistically significant results is problematic because it enhances the chances of false positive results and may inflate effect size estimates. In this review article, we present an extensive list of 34 degrees of freedom that researchers have in formulating hypotheses, and in designing, running, analyzing, and reporting of psychological research. The list can be used in research methods education, and as a checklist to assess the quality of preregistrations and to determine the potential for bias due to (arbitrary) choices in unregistered studies.

Keywords: bias; experimental design; p-hacking; questionable research practices; research methods education; significance chasing; significance testing.

PubMed Disclaimer

Cited by

Subjective evidence evaluation survey for many-analysts studies.
Sarafoglou A, Hoogeveen S, van den Bergh D, Aczel B, Albers CJ, Althoff T, Botvinik-Nezer R, Busch NA, Cataldo AM, Devezer B, van Dongen NNN, Dreber A, Fried EI, Hoekstra R, Hoffman S, Holzmeister F, Huber J, Huntington-Klein N, Ioannidis J, Johannesson M, Kirchler M, Loken E, Mangin JF, Matzke D, Menkveld AJ, Nilsonne G, van Ravenzwaaij D, Schweinsberg M, Schulz-Kuempel H, Shanks DR, Simons DJ, Spellman BA, Stoevenbelt AH, Szaszi B, Trübutschek D, Tuerlinckx F, Uhlmann EL, Vanpaemel W, Wicherts J, Wagenmakers EJ. Sarafoglou A, et al. R Soc Open Sci. 2024 Jul 24;11(7):240125. doi: 10.1098/rsos.240125. eCollection 2024 Jul. R Soc Open Sci. 2024. PMID: 39050728 Free PMC article.
Training Load and Its Role in Injury Prevention, Part 2: Conceptual and Methodologic Pitfalls.
Impellizzeri FM, McCall A, Ward P, Bornn L, Coutts AJ. Impellizzeri FM, et al. J Athl Train. 2020 Sep 1;55(9):893-901. doi: 10.4085/1062-6050-501-19. J Athl Train. 2020. PMID: 32991699 Free PMC article.
Does chief executive compensation predict financial performance or inaccurate financial reporting in listed companies: A systematic review.
Rousseau D, Kim BJ, Splenda R, Young S, Lee J, Beck D. Rousseau D, et al. Campbell Syst Rev. 2023 Dec 11;19(4):e1370. doi: 10.1002/cl2.1370. eCollection 2023 Dec. Campbell Syst Rev. 2023. PMID: 38089569 Free PMC article. Review.
The Hong Kong Principles for assessing researchers: Fostering research integrity.
Moher D, Bouter L, Kleinert S, Glasziou P, Sham MH, Barbour V, Coriat AM, Foeger N, Dirnagl U. Moher D, et al. PLoS Biol. 2020 Jul 16;18(7):e3000737. doi: 10.1371/journal.pbio.3000737. eCollection 2020 Jul. PLoS Biol. 2020. PMID: 32673304 Free PMC article.
The past, present and future of Registered Reports.
Chambers CD, Tzavella L. Chambers CD, et al. Nat Hum Behav. 2022 Jan;6(1):29-42. doi: 10.1038/s41562-021-01193-7. Epub 2021 Nov 15. Nat Hum Behav. 2022. PMID: 34782730 Review.

See all "Cited by" articles

References

1. Asendorpf J. B., Conner M., Fruyt F. D., Houwer J. D., Denissen J. J. A., Fiedler K., et al. (2013). Recommendations for increasing replicability in psychology. Eur. J. Personal. 27 108–119. 10.1002/per.1919 - DOI
1. Bakker M., Hartgerink C. H., Wicherts J. M., van der Maas H. L. (2016). Researchers’ Intuitions about power in psychological research. Psychol. Sci. 27 1069–1077. 10.1177/0956797616647519 - DOI - PMC - PubMed
1. Bakker M., van Dijk A., Wicherts J. M. (2012). The rules of the game called psychological science. Perspect. Psychol. Sci. 7 543–554. 10.1177/1745691612459060 - DOI - PubMed
1. Bakker M., Wicherts J. M. (2011). The (mis)reporting of statistical results in psychology journals. Behav. Res. Methods 43 666–678. 10.3758/s13428-011-0089-5 - DOI - PMC - PubMed
1. Bakker M., Wicherts J. M. (2014). Outlier removal, sum scores, and the inflation of the Type I error rate in independent samples t tests. The power of alternatives and recommendations. Psychol. Methods 19 409–427. 10.1037/met0000014 - DOI - PubMed

Publication types

Actions

LinkOut - more resources

Full Text Sources
Other Literature Sources
- scite Smart Citations

[1] Asendorpf J. B., Conner M., Fruyt F. D., Houwer J. D., Denissen J. J. A., Fiedler K., et al. (2013). Recommendations for increasing replicability in psychology. Eur. J. Personal. 27 108–119. 10.1002/per.1919 - DOI

[2] Asendorpf J. B., Conner M., Fruyt F. D., Houwer J. D., Denissen J. J. A., Fiedler K., et al. (2013). Recommendations for increasing replicability in psychology. Eur. J. Personal. 27 108–119. 10.1002/per.1919 - DOI

[3] Bakker M., Hartgerink C. H., Wicherts J. M., van der Maas H. L. (2016). Researchers’ Intuitions about power in psychological research. Psychol. Sci. 27 1069–1077. 10.1177/0956797616647519 - DOI - PMC - PubMed

[4] Bakker M., Hartgerink C. H., Wicherts J. M., van der Maas H. L. (2016). Researchers’ Intuitions about power in psychological research. Psychol. Sci. 27 1069–1077. 10.1177/0956797616647519 - DOI - PMC - PubMed

[5] Bakker M., van Dijk A., Wicherts J. M. (2012). The rules of the game called psychological science. Perspect. Psychol. Sci. 7 543–554. 10.1177/1745691612459060 - DOI - PubMed

[6] Bakker M., van Dijk A., Wicherts J. M. (2012). The rules of the game called psychological science. Perspect. Psychol. Sci. 7 543–554. 10.1177/1745691612459060 - DOI - PubMed

[7] Bakker M., Wicherts J. M. (2011). The (mis)reporting of statistical results in psychology journals. Behav. Res. Methods 43 666–678. 10.3758/s13428-011-0089-5 - DOI - PMC - PubMed

[8] Bakker M., Wicherts J. M. (2011). The (mis)reporting of statistical results in psychology journals. Behav. Res. Methods 43 666–678. 10.3758/s13428-011-0089-5 - DOI - PMC - PubMed

[9] Bakker M., Wicherts J. M. (2014). Outlier removal, sum scores, and the inflation of the Type I error rate in independent samples t tests. The power of alternatives and recommendations. Psychol. Methods 19 409–427. 10.1037/met0000014 - DOI - PubMed

[10] Bakker M., Wicherts J. M. (2014). Outlier removal, sum scores, and the inflation of the Type I error rate in independent samples t tests. The power of alternatives and recommendations. Psychol. Methods 19 409–427. 10.1037/met0000014 - DOI - PubMed

Save citation to file

Email citation

Add to Collections

Add to My Bibliography

Your saved search

Create a file for external citation management software

Your RSS Feed

Degrees of Freedom in Planning, Running, Analyzing, and Reporting Psychological Studies: A Checklist to Avoid p-Hacking

Affiliation

Degrees of Freedom in Planning, Running, Analyzing, and Reporting Psychological Studies: A Checklist to Avoid p-Hacking

Authors

Affiliation

Abstract

Similar articles

Cited by

References

Publication types

LinkOut - more resources

Full Text Sources

Other Literature Sources