Allen’s personal comments

Evaluating Research Sources

In a way, the problem is as old as universities: we’re trying to make new knowledge using building blocks supplied by others, so we need to evaluate—somehow—the information we have to work with. I begin with a couple of rules:

1. Absolutely everyone writes with bias. You can’t get around it. You see truth and the world in a certain way, and you want to get the rest of us to agree with you. There’s no such thing as a totally unbiased source, BUT you can recognize the built-in skewing in the authors you read and compensate for it.
2. Academic “truth” always amounts to the best guess based on the available information. You have to get into arbitrary fields like pure mathematics to encounter unchangeable rules that are always true for everyone everywhere and undeniably so. When you investigate anything in the physical world (even the law of gravity!) you are making guesses based on observations, and those observations are always susceptible to modification.
3. Even a terrible source might well have something of value in it. Rightly used, few sources are total poison. (OK, I can think of a few, but most serious scholarship has a place.)
4. Even a perfectly wonderful source might not work for your paper. “Perfect” has to contain the concept “perfect for the purpose you are trying to accomplish.”

Traditional Basics to Look For

Obviously, the more familiarity you have with a field, the easier this sort of discussion gets. This is why it’s a poor idea to do a research paper on a topic you are totally unfamiliar with.

Scope, Depth, and Obvious Audience

You will be tempted to go for the pre-digested work in Wikipedia or other tertiary sources¹. There’s nothing exactly wrong with those sources, but you need more. Particularly, those “information lite” sources act as if all the disputes were settled and all the questions answered. They are not. And you need to go deeper if you are going to do critical thinking. There’s nothing wrong with using “information lite” sources as a first word on something; just don’t assume they are the last word.

Yes, it’s work and it will make your head hurt. Make friends with the dictionary, and don’t be afraid to ask teachers for help. (That’s what they pay us for.)

Date

This criterion is a bit rubbery. If you’re working on a scientific paper, the most recent stuff is usually the best. Some truth, however, isn’t that date-sensitive. One example might be critical analysis of a short story by Poe: fifty year old material might be just fine. And obviously, if you are writing something about historical perspective, you’ll want something from the right era.

Who wrote it

This one is difficult, especially for undergraduates. Some sources, for example politicians and sites such as InfoWars, Breitbart, and Fox News are only interested in pushing their agenda, getting more viewers, and making people angry; you can’t learn anything truthful from them. Others are speaking outside their field of expertise. (One memorable example was the television evangelist who promoted a nutritional supplement; he may have known a lot about the Bible, but that didn’t say anything about his knowledge of nutrition.)

One quick question to ask is whether the source is anonymous (a red flag) and whether the source ever got edited. These aren’t water-tight tests. Some anonymous sources are excellent and fair, and some edited items are wildly biased. But it’s a beginning.

A common error is to assume that anything on the Internet is true. Not so! There is absolutely no filter that evaluates whether an Internet source is true—all you need to start writing something is a valid credit card and enough money to pay for a domain name and some server space. If you publish to Blogspot, Twitter, Facebook, or YouTube, you don’t even need that much; all you need is a way to borrow a computer. Nobody will ever check up on whether you are telling the truth.

Generally, you are on safer ground if you are working with information from major newspapers, news networks, universities, and book publishers. They have a vested interest in presenting credible information and protecting their reputations. (As an example, major newspapers such as the New York Times or Washington Post don’t publish an account of some event—earthquake, flood, or military attack—until they have four independent reports that agree.)

Bias and Pseudo-Science

Some of it is pretty obvious. You can just assume that any website that begins with Your Doctor is Lying to You! or Eat This Weird Fruit and Never Forget Anything Again! is off the edge of scientific truth. Nobody in the health industry has a vested interest in keeping you from the Miracle of Mangosteen Juice—so perhaps it won’t cure warts, cancer, impotence and/or stuttering.

Some bias is a lot harder to detect, particularly if they are pandering to your own bias. Generally, pseudo-science reveals itself with several trademarks:

• Whether the question is global warming, benefits of some diet supplement, or some theory of racial superiority, pseudo-science often makes a point of dismissively rejecting the work of regular scientists. Sometimes it’s a personal attack.
  • People who disagree with me don’t know anything.
  • People who disagree with me are trying to keep you in the dark.
  • Only I (or my group) have the truth.
• Pseudo-science often requires you to believe things that fly in the face of common sense.
  • For millions of years humanity has been yearning for this one obscure tropical berry that my company has an exclusive license to sell.
  • Coal-fired power plants produce almost zero CO₂
  • Everything that’s wrong with our country can be traced back to this one religious/ethnic group.
• Pseudo-science often begins with an agenda or cause to support, then invents “facts” to back themselves up. (Real science typically begins with an observation or discovery, then discusses the implications of it—not the other way around.)
• Sometimes pseudo-science works by bending the rules. They might ask the opposing view to provide the kind of water-tight proof that true science never gives; when they don’t get it, they make the point that “experts are uncertain, so in the absence of proof I’m right after all.” Examples would be the pro-smoking scientific publications (funded by tobacco companies) and much of the anti-global-warming writing².
• Pseudo-science is sometimes mis-labeled. They might be making a claim that gay people are inferior, but they present it as an issue of religious freedom; they might be trying to protect the profits of coal companies, but they present the problem as an attack on patriotism and freedom.
• Sometimes it’s quite difficult to determine who is behind a pseudo-science assertion. If your source is concealing information, that’s a big red flag.

The Community

This one is difficult for a beginning researcher like yourself. As a starting point, if several sources (not cherry-picked to support your point) agree, then you are on safer ground than if you find exactly one that is supporting an idea and everyone else is disagreeing. Yes, this is an argument for doing a research paper with several sources. If your big idea is only present in one reading, you should ask some serious questions.

A common error has gotten the humorous name Argumentum ad Google. It’s the error of assuming that a website with a higher Google rating is a better, more reliable source. Websites get higher Google ratings because they do a better job of managing subheads (such as my The Community subhead), repeat key words, have a lot of “good” links in and out, and (sometimes) just pay Google some money.

1. A primary source is something written by the researcher who originally did the work (for example, the person who counted the number of smokers who got lung cancer). A secondary source is a piece written by researchers who studied several primary sources and wrote a paper that draws conclusions from those sources (for example, the 1964 Surgeon General’s report that tied smoking to increased lung cancer, emphysema, and a number of other diseases). A tertiary source is something like Wikipedia or a popular magazine or newspaper that retails the results for non-experts.
2. One example of this kind of rule-bending is misuse of the word “theory.” In informal chatter over the barbecue, “Well, that’s just your theory” means “You don’t know what you are talking about and you invented that.” The academic definition of “theory” is “a system of ideas intended to explain something,” so we can talk about the “theory of gravity,” even though we are pretty certain that gravity will always work. Another example is misuse of the word “proof.” In the real world, you almost never get the kind of proof that mathematics offers: A² + B² = C². You get educated guesses based on available information: The Pfizer Covid-19 vaccine is 95% effective in preventing infection, so you will probably not get the disease if you are vaccinated. We don’t get 100% certainty.

The views and opinions expressed in this page are strictly those of the page author.
The contents of this page have not been reviewed or approved by Ashland University.
Revised 8/6/21 • Page author: Curtis Allen • e-mail: callen@ashland.edu.