Part 1 of a 4-Part Series
- View Part 2 here: Peer Review, Reproducibility: How To Separate Good Science From Sketchy Tales
- View Part 3 here: Bad Science Thrives When Corporations & Government Regulators Deny Peer Review
- View Part 4 here: Private Science For Hire: Poster Children For Un-trustworthy Science
To paraphrase Winston Churchill, scientific peer review is the worst system in the world … except for all the rest. (Scroll to the bottom of this article for his exact quote which applied to democracy.)
Fortunately, peer review works almost all the time.
A massive database of retracted papers released in 2018 indicates that only two of every 10,000 papers has ever been retracted. Of course, that counts only those people who got caught.
Undoubtedly, there are other undetected perpetrators, but even a 10-fold increase would mean that peer review keeps fraudulent papers down to 0.2 percent.
What’s peer review”
In short, peer review requires a scientific paper being considered for publication to be submitted to experienced scientists (without conflicts of interest) who are knowledgeable about the topics in the paper. Conflicts of interests must be disclosed by any of the authors of a paper.
The peer reviewers make comments about the way the experiment was designed, the protocols involved, the quality of the data, whether the data supports the papers conclusions and other factors that determine whether the paper is suitable for publication.
In addition, properly selected peer reviewers often know of undisclosed biases and conflicts of interest among paper authors that need further disclosure.
Often a paper is edited or re-written to address comments before publication. Sometimes the flaws are too severe and the paper is rejected. Sadly, the rejected paper may eventually be published by a journal with sketchy ethics.
When it works correctly, peer review helps assure the overall quality of the article. Today, that has been combined with with more recent requirements by top scientific journals that papers include complete information sufficient for others to reproduce the experiment and results.
It addition to better science, this process helps protect a scientific journal’s reputation and credibility.
Systemic Flaws in Peer Review
As Churchill hinted, democracy is a messy system with systemic flaws. Peer review is no different.
Peer review has flaws, but it remains the best system to control the quality of science.
Sadly, corporate, private and regulatory science lacks any quality control system at all. See “Why You Can’t Trust Corporate & Federal Regulatory Science” for details.
The most serious systemic problem facing the peer review system is that scientific journals who charge hundreds and even thousands of dollars for an annual subscription pay their peer reviewers nothing.
Some reviewers work for free so they can get into the good graces of editors in hopes of getting their own papers reviewed.
Often, the most qualified reviewers are simply too busy with their work and cannot afford the time or the financial sacrifice to review another paper.
- As numbers of published articles rise, the scholarly review system must adapt to avoid unmanageable burdens and slipping standards
Perhaps an even worst systemic flaw is the practice of journal editors asking a prospective author for a list of potential reviewers. Few scientists would stack that list with their enemies.
That can produce the same results as a crooked science fair
Conflict Of Interest Disclosure
While respectable journals require conflict of interest disclosures of authors, they do not require the same of reviewers.
In addition, journals use a very narrow definition of “conflict of interest” to mean a direct financial interest.
Neither reviewers nor journal authors are required to disclose previous associations such as employment or consulting positions with chemical corporations, tobacco companies, pharmaceutical and medical manufacturers, private laboratories, advocacy and leadership positions with industry groups and other parties which have direct financial interests in the outcome of studies and their conclusions.
A Major Fraudulent Flaw
While the narrow definition of “conflict of interest” is an inherent problem that journals are trying to address, outright deception and fraud are possible. Examples of this can be found at:
- Toxic Science: Fiddling Facts For Profit & Legal Leverage
- Private Peer Review Like A Crooked Science Fair
In addition, another example of fraud that is being exposed lies in networks of fake reviewers — people who simply do not exist. A number of those have recently resulted in the retraction of scores of scientific papers. (See: Scholarly journal retracts 60 articles, smashes “peer review ring”)
According to the Washington Post: “a major publisher of scholarly medical and science articles has retracted 43 papers because of “fabricated” peer reviews amid signs of a broader fake peer review racket affecting many more publications.
[T]he Committee on Publication Ethics, a multidisciplinary group that includes more than 9,000 journal editors, issued a statement suggesting a much broader potential problem. The committee, it said, “has become aware of systematic, inappropriate attempts to manipulate the peer review processes of several journals across different publishers.” Those journals are now reviewing manuscripts to determine how many may need to be retracted, it said.
Keeping An Eye On Sketchy Science
Retraction Watch is a magnificent source of information on sketchy science.
- Mistakes are part of science. But setting the record straight promptly and clearly can help to avoid a career blot.
- As numbers of published articles rise, the scholarly review system must adapt to avoid unmanageable burdens and slipping standards, says
|Sir Winston Leonard Spencer Churchill (1874–1965)|
|QUOTATION:||Many forms of Government have been tried, and will be tried in this world of sin and woe. No one pretends that democracy is perfect or all-wise. Indeed, it has been said that democracy is the worst form of Government except all those other forms that have been tried from time to time.|
|ATTRIBUTION:||WINSTON CHURCHILL, speech, House of Commons, November 11, 1947.—Winston S. Churchill: His Complete Speeches, 1897–1963, ed. Robert Rhodes James, vol. 7, p. 7566 (1974).|