How to spot bad science

by

110420tedtalks-500x291The TED brand, run by the folks who hold annual conferences to spread good ideas, have some truly inspiring clips (available for free). If you are not familiar with them, go check a few out, you will not be disappointed, and as a tip for finding the cream, I’d recommend that you crowd-source and pick the most popular. They are all quite digestible because each of the invited speakers get an 18 minute slot, so this forces them to not babble, but instead distil their thinking.

As an extension to TED, they have a franchise called TEDx where they grant licenses to third parties to organize independent TED-like events internationally. This has proven to be very popular because, as reported in June 2012, five TEDx events on average were being organized every day, in one of 133 countries. Rather sadly in some instances, these third parties have been damaging the TED brand by propagating speakers who promote complete bullshit.

Saving the TED Brand

In order to address this rising tide of concern that is being raised in public forums, the TEDx directors (Lara Stein, TEDx Director and Emily McManus, TED.com Editor) have written an open letter to TEDx organizers advising them on how to avoid promoting bad science. This letter is a really good and contains guidance that is of interest, not just to TEDx organizers, but to anybody interested in working out what is good and what is in fact complete bullshit – and that means you and me.

The letter covers 3 topics …

  1. A short definition of bad science / pseudoscience.
  2. Common warning signs of bad science and health hoaxes — above and beyond the science itself — how can you spot trouble?
  3. Topics to watch out for, because in the past they have attracted bad science to TEDx events

Lets take a look …

What is bad science / pseudoscience

here are some basic guidelines.

Marks of good science:

  • It makes claims that can be tested and verified
  • It has been published in a peer reviewed journal (but beware… there are some dodgy journals out there that seem credible, but aren’t.)
  • It is based on theories that are discussed and argued for by many experts in the field
  • It is backed up by experiments that have generated enough data to convince other experts of its legitimacy
  • Its proponents are secure enough to accept areas of doubt and need for further investigation
  • It does not fly in the face of the broad existing body of scientific knowledge
  • The proposed speaker works for a university and/or has a phD or other bona fide high level scientific qualification

Marks of bad science:

  • Has failed to convince many mainstream scientists of its truth
  • Is not based on experiments that can be reproduced by others
  • Contains experimental flaws or is based on data that does not convincingly corroborate the experimenter’s theoretical claims
  • Comes from overconfident fringe experts
  • Uses over-simplified interpretations of legitimate studies and may combine with imprecise, spiritual or new age vocabulary, to form new, completely untested theories.
  • Speaks dismissively of mainstream science
  • Includes some of the red flags listed in the two sections below

Warning signs of bad science and health hoaxes

Food science, including:

  • GMO food and anti-GMO foodists
  • Food as medicine, especially to treat a specific condition: Autism and ADHD, especially causes of and cures for autism

Because of the sad history of hoaxes with deadly consequences in the field of autism research, really look into the background of any autism-related talk. If you hear anything that sounds remotely like, “Vaccines are related to autism,” — RUN AWAY. Another non-legitimate argument: “We don’t know what works, so we have to try everything.” Pretty much all the time, this argument is designed to cause guilt in suffering parents so they’ll spend money on unproven treatments.

More: 

  • “Healing,” including reiki, energy fields, alternative health and placebos, crystals, pyramid power
  • “Free energy” and perpetual motion machines, alchemy, time travel
  • The neuroscience of [fill in the blank] — not saying this will all be non-legitimate, but that it’s a field where a lot of goofballs are right now
  • The fusion of science and spirituality. Be especially careful of anyone trying to prove the validity of their religious beliefs and practices by using science

Topics to watch out for

Be alert if a potential speaker (or the speaker’s advocate on your planning team) does any of the following things:

  • Barrages you with piles of unrelated, over-general backup material, attempting to bury you in data they think you won’t have time to read
  • Holds a nonstandard degree. For instance, if the physics-related speaker has a degree in engineering, not physics; if the medical researcher does not have an M.D. or Ph.D.; if the affiliated university does not have a solid reputation. This is not snobbery; if a scientist truly wishes to make an advance in their chosen field, they’ll make an effort to engage with other scholars
  • Claims to have knowledge no one else has
  • Sends information only from websites they created themselves; there is little or no comment on them in mainstream science publications or even on Wikipedia
  • Provides data that takes the form of anecdotes, testimonials and/or studies of only one person
  • Sells a product, supplement, plan or service related to their proposed talk — this is a BIG RED FLAG
  • Acts oddly persistent about getting to your stage. A normal person who is rejected for the TEDx stage will be sad and usually withdraw from you. A hoaxer, especially one who sees a financial upside to being associated with TEDx, will persist, sometimes working to influence members of your team one by one or through alternative channels
  • Accuses you of endangering their freedom of speech. (Shutting down a bogus speaker is in no way endangering their freedom of speech. They’re still free to speak wherever they can find a platform. You are equally free not to lend them the TEDx platform.)
  • Demands that TEDx present “both sides of an issue” when one side is not backed by science or data. This comes up around topics such as creationism, anti-vaccination and alternative health
  • Acts upset or hurt that you are checking them out or doubting them
  • Accuses you of suppressing them because TED and TEDx is biased against them and run by rich liberals ;)
  • Threatens to publicly embarrass TED and TEDx for suppressing them. (The exact opposite will happen.)

Links

Leave a Comment