Send us a link
Standard reports paint a much rosier picture of the research landscape than may be warranted. In this analysis, the first hypothesis of standard articles reported was supported by the data 96% of the time, while that rate was only 44% in registered reports.
Kvarven, Strømland and Johannesson compare meta-analyses to multiple-laboratory replication projects and find that meta-analyses overestimate effect sizes by a factor of almost three. Commonly used methods of adjusting for publication bias do not substantively improve results.
A new study found that Registered Reports are only about 50% as likely as standard, non-RR research to confirm their hypothesis.
Science is messy, and the results of research rarely conform fully to plan or expectation. ‘Clean’ narratives are an artefact of inappropriate pressures and the culture they have generated.
PLOS ONE will soon offer Registered Reports, a preregistration option which enables open peer review and publication of the initial study protocol in advance of the full research article.
The question whether the practice of preregistration is valuable depends on your philosophy of science. Daniel Lakens provides a conceptual analysis of the value of preregistration for psychological science from an error statistical philosophy.
There is little reason to expect that preregistration will spontaneously help researchers to develop better theories (and, hence, better methods and analyses).
Reviewing and accepting study plans before results are known can counter perverse incentives. Chris Chambers sets out three ways to improve the approach.
Making decisions before conducting analyses requires practice. Respecting both what was planned and what actually happened requires good judgment and humility in making claims. With the accelerating adoption of preregistration, we now face the challenge of figuring out how to use this methodology to its fullest potential.
The research community needs to find ways to reward study design and methodology as much as the final result. A publishing format called Registered Reports offers a means of addressing this challenge.
So, as the Statistical War and Tone War are in a lull, the Preregistration conflict has flared up yet again.
A blog about the science of human behavior and the human behavior of scientists.
