Let’s not fool ourselves: The importance of battling questionable research practices (QRPs)

In the past 5–10 years, many scientific disciplines have come under fire for shoddy research practices, perverse incentives, and a lack of transparency and openness. If you want to do solid and meaningful science, then it’s vital to not fool yourself as part of the research process. To minimize the chances of misleading yourself, being misled, and mistakenly believing in effects that in fact are spurious, we highly recommend that you read on!

One example for how bad it can get can be taken from psychology. In an article published in 2011, Simmons, Nelson, and Simonsohn (2011) have raised concerns about researcher degrees of freedom and the extent to which they can increase the chances of false-positive findings. Amongst others, Simmons and colleagues reported a finding that listening to a Beatles song made their participants become younger – their age supposedly decreased! This impossible finding only became possible after engaging in what is called questionable research practices (QRPs).

More precisely, in the course of collecting and analyzing data, researchers have the option to flexibly make a range of decisions - whether or not to collect more data, to exclude some observations, to include control variables, to combine or transform measures, or to report only a subset of experimental conditions. Given this flexibility, it is all too alluring to dredge your data until you finally find a significant effect. Simmons et al. (2011) discovered that these researcher degrees of freedom “made it unacceptably easy (...) to accumulate (and report) statistically significant evidence for a false hypothesis” and to “present anything as [statistically] significant” (p. 1359). For a similar discussion of the problems with data-dependent analysis, or the ‘garden of forking paths’, see this great article by Gelman and Loken (2014).

So, what can we do about QRPs? Read: How can me make science more robust and reliable?

Need further help?
Click here to contact us