False-Positive Psychology: Undisclosed Flexibility in Data Collection and Analysis Allows Presenting Anything as Significant | Joseph P. Simmons et al. | ResearchPod

Transcript: False-Positive Psychology: Undisclosed Flexibility in Data Collection and Analysis Allows Presenting Anything as Significant

[[RP_SECTION:researcher-degrees-of-freedom|Researcher Degrees of Freedom]]

Sam: [steady, matter-of-fact] The nominal five percent false-positive rate is a statistical fiction. Undisclosed flexibility in data analysis allows researchers to reach significance more than sixty percent of the time. That is the core conclusion of a 2011 paper by Simmons, Nelson, and Simonsohn.

Alex: [curious, leaning in] Sixty percent? If alpha is set at point-zero-five, how does the error rate balloon that dramatically?

Sam: [measured, teaching mode] It happens through what the authors call "researcher degrees of freedom." Think of it like a radio tuner. If you scan the dial, you will eventually find a station, even if the airwaves are empty. In research, those dial turns are decisions like when to stop collecting data, which outliers to drop, or which covariates to include. Each choice might seem defensible in isolation, but the combination creates massive, uncontrolled inflation of the Type I error rate.

Alex: [processing] So it is not necessarily malice — it is iteratively testing different analytic paths until you hit that p-less-than-point-zero-five threshold.

Sam: [precise] Exactly. And to prove the point empirically, the authors published a demonstration study showing that listening to The Beatles makes you chronologically younger. They used nothing but legitimate statistical practices — just exploited these degrees of freedom to manufacture a significant result where none existed.

Alex: [slower, reflecting] That is a sobering demonstration. If the path to significance is paved with post-hoc choices, the p-value loses its diagnostic power entirely.

Sam: [grounded, serious] That is precisely the point. And the publication incentive structure makes it worse. Because null results are rarely published, researchers feel pressure to find significance. This creates a feedback loop where analytic choices get justified by the results they produce. Combine just four common degrees of freedom — sample size flexibility, covariate inclusion, outlier exclusion, and condition selection — and your false-positive rate jumps from five percent to over sixty percent.

Alex: [thoughtful] So the problem is fundamentally one of transparency. If we cannot see the analytic path the researcher took, we have no way to know whether the result is robust or just a product of selective analysis.

Sam: [building the case] Precisely. The authors propose a disclosure-based solution. Researchers should be required to report all variables collected, all conditions run, and all results — including those that did not make the final paper. Forcing that transparency makes it much harder to hide the dial-turning that produces these false positives.

False-Positive Psychology: Undisclosed Flexibility in Data Collection and Analysis Allows Presenting Anything as Significant

Joseph P. Simmons, Leif D. Nelson, Uri Simonsohn

Oct 17, 2011·5:57·pmc·Original Paper

False-Positive PsychologyResearcher Degrees of FreedomP-hackingResearcher degrees of freedomFalse-positive rateReplication crisis

Listen

6 min

Briefing

The Problem of Researcher Degrees of Freedom

In empirical psychology, the standard for statistical significance is typically a p-value of 0.05 or less. However, the authors argue that this threshold is effectively meaningless when researchers have excessive flexibility in how they collect, analyze, and report their data. This flexibility—referred to as researcher degrees of freedom—includes decisions such as when to stop collecting data, which variables to include or exclude, and which covariates to use. Because researchers are often motivated to find significant results, they may unconsciously (or consciously) explore various analytic paths until they find one that yields a p-value below the 0.05 threshold, reporting only that successful path.

Demonstrating the Ease of False Positives

To illustrate how easily this flexibility can lead to false-positive findings, the authors conducted two experiments designed to show that listening to certain songs can change a person's age. By selectively choosing which variables to report and using specific covariates, they were able to produce statistically significant results for these absurd hypotheses. They supplemented these experiments with computer simulations, which showed that combining common, seemingly minor analytic choices can increase the actual false-positive rate from the nominal 5% to as high as 61%.

A Disclosure-Based Solution

To combat this, the authors propose a set of six requirements for authors and four guidelines for reviewers. The core of the solution is transparency: authors must disclose their data collection termination rules, report all variables collected, and provide results both with and without covariates or data exclusions. The goal is not to eliminate exploratory research, but to ensure that such research is clearly labeled as such and that readers can distinguish between robust findings and those that rely on arbitrary, post-hoc analytic decisions.

Key Terms and Definitions

Researcher Degrees of Freedom — The various, often undisclosed choices researchers make during data collection and analysis that can be manipulated to achieve statistical significance.
False Positive — The incorrect rejection of a null hypothesis, leading to the false conclusion that an effect exists when it does not.
Ex Ante — Decisions or rules established before data collection begins, which the authors argue are necessary to prevent post-hoc manipulation of results.
File-Drawer Problem — The tendency for researchers to only publish studies that yield significant results, while leaving null results in their "file drawers," which biases the scientific literature.
Confirmatory Research — Research conducted to test a specific, pre-registered hypothesis, which the authors contrast with exploratory research that searches for patterns in data.

Abstract

In this article, we accomplish two things. First, we show that despite empirical psychologists' nominal endorsement of a low rate of false-positive findings (≤ .05), flexibility in data collection, analysis, and reporting dramatically increases actual false-positive rates. In many cases, a researcher is more likely to falsely find evidence that an effect exists than to correctly find evidence that it does not. We present computer simulations and a pair of actual experiments that demonstrate how unacceptably easy it is to accumulate (and report) statistically significant evidence for a false hypothesis. Second, we suggest a simple, low-cost, and straightforwardly effective disclosure-based solution to this problem. The solution involves six concrete requirements for authors and four guidelines for reviewers, all of which impose a minimal burden on the publication process.

False-Positive Psychology: Undisclosed Flexibility in Data Collection and Analysis Allows Presenting Anything as Significant: Research Paper Podcast