Part 2 of a 4-Part Series
- View Part 1 here: What is Peer Review And How It Can Go Wrong?
- View Part 3 here: Bad Science Thrives When Corporations & Government Regulators Deny Peer Review
- View Part 4 here: Private Science For Hire: Poster Children For Un-trustworthy Science
To a non-scientist, a study is a study is a study. All seem equally complex, dense, and resistant to understanding for intelligent non-scientific laypersons.
What’s more, the science that affects our everyday lives and health has become so complicated and increasingly specialized that a scientist in one field cannot be expected to understand a study in another discipline.
Complicating this already baffling situation are studies that may reach different conclusions about the same issue (coffee is bad; coffee is good; enough coffee is good but not too much or too little…and what is too much or too little?)
And as if this wasn’t enough, consumers are forced to make significant health and lifestyle decisions based on incomplete data gleaned from headlines and articles usually written by journalists who don’t understand the science much better than their non-scientist readers or viewers.
The Quest For Certainty
The biggest problem is that science is a process and life is a decision.
Results from a single study rarely merit a definitive conclusion on any important personal decision.
While there are exceptions, the results from a single study report are valid for what happened in that experiment and under a specific set of circumstances and a specific set of chemicals, measurements, animals, procedures and other variables.
The findings of a study can vary because of how a study is designed … whether it looks for the same information in the same or different ways. And whether the study was conducted by those with conflicts of interest in which they have a stake in the outcome of the results.
For all those reasons, important decisions should not be based on a single study, but need to be made on the basis of a consensus of the overall body of trustworthy studies considered as a whole.
In the middle of all this ambiguity, the reality of everyday life demands that people resolve uncertainty unto certainty by making decisions: buy this product, not that one. Eat this, not that.
In the end, the uncertainty and complexity force many to base their decisions on factors that reinforce already-held beliefs and on near, anecdotes, personal narratives and “I heard it somewhere” stories from friends and family.
(To dig deeper on this, see: Americans Believe in Science, Just Not Its Findings and other links at the bottom of this article).
What Can You Trust? Who?
But what … or who to trust?
Fortunately, you can start to make sense of which studies to trust without understanding the science itself.
This is because trustworthy studies have a number of solid, easy to detect safeguards that separate them from sketchy tales.
The most trustworthy science is:
- Peer reviewed,
- Conducted by scientists without conflicts of interest, and
- Reproducible by other scientists.
It’s important to understand that those three elements do not guarantee a perfectly trustworthy study. But they offer safeguards that can blow the whistle on misconduct, dishonesty, bias, and incompetence.
Blank Checks & No Receipts: A Grounds For Mistrust
Trusting a scientific study without those safeguards is like going to the grocery store checkout stand, handing the clerk a signed, blank check, then closing your eyes.
When everything is done, the clerk hands you your bags. No receipt, just the amount of the check they have filled out.
The clerk says “trust me” that the amount is correct, not inflated, and that the bags have all the items you have paid for.
Honest mistakes can be made. Dishonest acts can be hidden. Without transparency and verification, trust means very little.
Sadly, these scientific trust principles are almost never followed for studies done by private and corporate labs for studies that are used by the federal government for regulatory approvals.
While it’s vital to realize that good science can be often be found in studies that are not peer-reviewed, they do not merit your trust because their results and data cannot be examined or reproduced by independent third parties.
More on that later, but it’s important to focus now on what can be trusted so you can easily recognize those studies that fail.
What’s Involved With Peer Review?
Before publishing a paper, respected peer-reviewed journals send it to a panel of other scientists scrutinize the paper, its data, experiment design, methods and conclusions before it’s recommended for publication.
In addition, every respectable journal requires that all of the authors disclose any conflicts of interest.
Once a paper is published in a peer-reviewed journal, it becomes open to scrutiny by the entire scientific world.
The resulting comment letters often point out flaws in design, or perhaps note that data has been hand-picked to exclude items that run counter to the published conclusion.
Comment letters often point out flaws in design, or perhaps note that data has been hand-picked to exclude items that run counter to the published conclusion. Based on these letters and comments, papers are sometimes amended, corrected or — in the extreme — retracted entirely.
As we will see, below, peer review has its flaws and potential for abuse. Its protections against abuse need strengthening, but are the best method that exists to date.
For that reason, you should not trust a study if it is not peer-reviewed in a respected scientific journal. Period. The science may possibly be good. But if the people behind a non-peer-reviewed study are unable to withstand the transparency and disclosure of peer review, then its conclusions remain suspect.
This ad-free article is made possible by the financial support of the
Center for Research on Environmental Chemicals in Humans: a 501(c)(3) non-profit.
Please consider making a tax-deductible donation for continued biomedical research.
Reproducibility & Transparency: Keeping Peer Review Honest
Despite peer-review’s basic safeguards, it can be gamed in subtle ways.
But sometimes, personal ambition or corporate profits turn science on its head. Pursuit of glory, a Nobel Prize or university tenure sometimes prompts an academic researcher to falsify data or slant the way an experiment is designed in order to produce a pre-conceived “breakthrough.”
Likewise, in the corporate and governmental worlds, studies are sometimes carefully crafted, slanted and biased in pursuit of regulatory approval or in pursuit of an agency or political goal.
In case like these, reproducibility is the best safeguard against unethical scientific chicanery.
Reproducibility demands that a study reveal every bit of information and details needed for another lab to reproduce the experiment. This means the design, all of the data, protocols, methods and materials — right down to the brand names of chemical reagents, equipment, and processes.
The National Institutes of Health and other prominent research-funders are now looking at ways to increase the number of studies that are reproduced.
Reforms will be tough because reproducing someone else’s “breakthrough” study is thankless, time-consuming, expensive, hard to fund … and there is no Nobel, no glory and little tenure for it.
That is why many of those who set out to reproduce a study are rivals setting out to debunk it. The motive may be less than noble, but the results do help keep science honest.
In any event, scammers are unlikely to publish the necessary information to comply with reproducibility because they are aware that it could be the evidence to take them down.
As the next section shows, reproducibility is a safeguard against both scammers who falsify data and the honest who have made an inadvertent error.
Reproducibility: The Gold Standard
In addition to proper peer review, credible scientific journals also require a paper’s authors to make every small detail and fact — data, methods etc. — available to scientists who want to reproduce the study’s conclusions.
A recent article in the Economist: “When science gets it wrong: Let the light shine in” highlighted two very prominent examples from 2014:
- A ground-breaking paper on how stem cells could be easily produced, and
- The first solid proof of the cosmological inflation theory of how the universe was created.
The details are in the link above, but briefly:
- The stem cell paper had to be retracted and the scientists involved censured when other scientists found that data had been improperly manipulated.
- The proof of cosmological inflation — which had actually been announced before publication — was refuted in a paper that concluded the antenna used to gather the data had been contaminated by space dust.
The cosmologists were not accused of unethical behavior as the stem cell researchers were, but it pointed out the self-correcting nature of peer review.
What Can Go Wrong With Peer Review?
While still the best method developed so far, the peer-review system does have its own loopholes — something top journals are in the process of closing.
For example, top scientific journals are currently considering a move to conceal the authorship of paper from reviewers. This would help avoid personal biases or those affected by “celebrity” scientists. For More, see: “Quality control in science journals is evolving, with a code of ethics in hot pursuit,” and “Nature journals offer double-blind review.
The issue of conflicts of interest is also one that the most respected journals are trying to address. The biggest loophole is that “conflict of interest” has mostly been narrowly defined as current or recent financial interests such as stock ownership or a direct monetary gain to be had by the outcome of the study.
This does not address issues of other forms of bias such as former employment by a company to be affected by the study outcome, or the promise of a job to come.
Also not addressed are ideological and political biases and previous involvement in deliberately biased studies designed to benefit a specific organization. A prime example of that would be the many scientists who were paid millions by the tobacco industry to produce junk science designed to hide or deny the health dangers of smoking.
Ideally, the proposed paper will also be peer-reviewed by scientists without their own conflicts of interest.
However, less-trustworthy journals have editors known to be friendly to certain industries or scientists.
Those journals quietly send papers to “old boy” reviewers who will rubber-stamp papers the journal favors and reject others.
UN-Trustworthy Studies Dominate Science
In deciding whom to trust, people should keep in mind that peer-review, conflict of interest disclosures and other safeguards are not required and, therefore, do not exist for the studies done by private labs, corporate labs and the science submitted to the federal government for regulatory approvals.
That will be the topic of a future post, but the following will offer a detailed look at a broken system of regulatory science that has made no substantial advancements since its first publication:
“Equal Treatment for Regulatory Science: Extending the Controls Governing the Quality of Public Research to Private Research,” American Journal of Law & Medicine, 30 (2004): 119-54, 2004 American Society of Law, Medicine & Ethics Boston University School of Law, by Wendy Wagner & David Michaels
Not Perfect, But The Best Safeguard Devised So Far
No system, including peer review, is perfect. It is, as Winston Churchill said of democracy, “The worst system in the world … except for all the rest.”
The issues over government and regulatory agency science, however, has no credible third-party scrutiny and is subject to a vast array of political and financial conflicts of interest.
Scientific Darwinism: One Final Consideration
There is a Darwinian element to scientific research.
My experience as a top aide on Capitol Hill and an investigative reporter in Washington has shown me that, while there are exceptions, those in the government and in corporations are well-paid and their labs funded by their employers.
Those who wish to keep their jobs in those government and corporate positions must keep their employers happy. They have few scientific peers whose opinions count other than to stand together with in an old-boy network.
The old-boys are no longer pressured by publish or perish. Few stay on the cutting edge of science because meeting bureaucratic and corporate goals, and billing as many clients as possible all trump academic studies and excellence.
Additionally, my experience as a scientist and a university faculty member has shown me that academic researchers have to compete tooth-and-nail, fighting for every last grant and making every dollar count for a study that has got to be accurate, reproducible and come through peer review intact. Those grants go to those with the best published papers based on the most advanced science.
Peer review and reproducibility are science’s gold standards and only one group of researchers must brave heaven and hell to meet those standards.
Regulatory agencies such as the FDA and EPA are lax on safeguards because the science they accept do not need to meet the standards of peer review, reproducibility and conflict of interests.
Within the federal government, only the National Institutes of Health holds researchers the highest standards for academic, scientific and experimental excellence.
Comments are closed.