Small is beautiful: In defense of the small-N design

In response to the ongoing replication crisis in psychology, many researchers have advocated for larger sample sizes (large-N designs) to improve statistical power. Smith and Little argue that this focus on sample size is misguided. They contend that the replication crisis stems not from small samples, but from a lack of strong measurement, weak theoretical models, and poor control over error variance. By shifting the focus from estimating population parameters to understanding functional relationships at the individual level, small-N designs offer a more rigorous path toward a mature, quantitative science.

Small-N designs treat the individual participant as the primary unit of replication. By collecting a large number of observations from a few individuals, researchers can achieve high measurement precision and test complex, process-oriented theories. This approach avoids the pitfalls of aggregating data across heterogeneous individuals, which can obscure important qualitative differences and lead to misleading conclusions. The authors highlight that fields like vision science, which frequently employ small-N designs, have largely avoided the replication crisis precisely because their methods prioritize precise, replicable measurements on individual observers.

To demonstrate the superiority of individual-level analysis, the authors simulated an additive factors experiment. Their results show that group-level analysis often masks individual differences, leading to low power and inferential indeterminacy unless sample sizes are extremely large. In contrast, individual-level analysis remains sensitive to the underlying cognitive processes even with very few participants. When individual-level effects are tested with high power, each participant serves as an independent replication, providing a built-in check on the reliability of the findings.

Smith and Little do not argue that large-N designs are never useful, but they caution against the trend of mandating large samples as a universal solution. They suggest that the field should prioritize methods that align with the theoretical level of the research question. For process-oriented research, small-N designs—supported by strong mathematical models—offer a more efficient and informative allocation of scientific resources.