The prevalence of statistical reporting errors in psychology (1985-2013)
- PMID: 26497820
- PMCID: PMC5101263
- DOI: 10.3758/s13428-015-0664-2
The prevalence of statistical reporting errors in psychology (1985-2013)
- PMID: 26497820
- PMCID: PMC5101263
- DOI: 10.3758/s13428-015-0664-2
Abstract
This study documents reporting errors in a sample of over 250,000 p-values reported in eight major psychology journals from 1985 until 2013, using the new R package "statcheck." statcheck retrieved null-hypothesis significance testing (NHST) results from over half of the articles from this period. In line with earlier research, we found that half of all published psychology papers that use NHST contained at least one p-value that was inconsistent with its test statistic and degrees of freedom. One in eight papers contained a grossly inconsistent p-value that may have affected the statistical conclusion. In contrast to earlier findings, we found that the average prevalence of inconsistent p-values has been stable over the years or has declined. The prevalence of gross inconsistencies was higher in p-values reported as significant than in p-values reported as nonsignificant. This could indicate a systematic bias in favor of significant results. Possible solutions for the high prevalence of reporting inconsistencies could be to encourage sharing data, to let co-authors check results in a so-called "co-pilot model," and to use statcheck to flag possible inconsistencies in one's own manuscript or during the review process.
Keywords: False positives; NHST; Publication bias; Questionable research practices; Reporting errors; Significance; p-values.
Figures
Fig. 1
The percentage of articles with…
Fig. 1
The percentage of articles with American Psychological Association (APA)-reported null-hypothesis significance testing (NHST)…
Fig. 2
The average number of American…
Fig. 2
The average number of American Psychological Association (APA)-reported null-hypothesis significance testing (NHST) results…
Fig. 3
The average percentage of articles…
Fig. 3
The average percentage of articles within a journal with at least one (gross)…
Fig. 4
Average percentage of inconsistencies (open…
Fig. 4
Average percentage of inconsistencies (open circles) and gross inconsistencies (solid circles) in an…
Fig. 5
Percentage of articles with at…
Fig. 5
Percentage of articles with at least one inconsistency (open circles) or at least…
Fig. 6
The percentage of gross inconsistencies…
Fig. 6
The percentage of gross inconsistencies in p -values reported as significant (white bars)…
Fig. 7
The percentage of gross inconsistencies…
Fig. 7
The percentage of gross inconsistencies in p -values reported as significant (solid line)…
Fig. 8
The total number of downloaded…
Fig. 8
The total number of downloaded articles and the number of published articles that…
Fig. 9
The average number of exact…
Fig. 9
The average number of exact and inexact null-hypothesis significance testing (NHST) results per…
Comment in
-
Stat-checking software stirs up psychology.Nature. 2016 Nov 25;540(7631):151-152. doi: 10.1038/540151a. Nature. 2016. PMID: 27905454 No abstract available.
References
-
- American Psychological Association . Publication Manual of the American Psychological Association. 3. Washington, DC: American Psychological Association; 1983.
-
- American Psychological Association . Publication Manual of the American Psychological Association. 6. Washington, DC: American Psychological Association; 2010.
MeSH terms
LinkOut - more resources
Full Text Sources
Other Literature Sources
