The Online Photographer

This week's column by Ctein

It's Column 300 (best as I can count), and as is my tradition I'm going off-topic into the realm of science. This time I'm going meta, but that will probably be of more practical value to most of you than most of my science columns.

As a layperson, how can you distinguish Good Science from Bad Science? If one source of information says "X" and another "not-X" how do you decide which one to believe when it's not a discipline that you know well enough to be able to evaluate on scientific merits? There are tools that'll let you make the right decision almost every time.

Please understand that "good" and "bad" aren't the same as "right" and "wrong." Bad science is almost always wrong, at least 99% of the time, so it's good to be able to sniff it out. But, on occasion, it proves correct. There are a few historical examples (Newton's theory of gravitation). Conversely, Good Science, while usually right, is regularly proven wrong (last year's report of faster-than-light neutrinos); that's part of the point of science. Picking the Good Science doesn't guarantee that you're correct, but picking the Bad Science almost always guarantees that you're not.

Neither Good nor Bad Science can tell you what is true. General relativity and quantum mechanics are Good Science. Are they actually true? We have no idea. Given the history of science, probably not. But they're correct enough, as demonstrated by your GPS (which makes use of GR), and your personal computers and electronics (which depend on QM).

Hundreds of you will be offended by these columns. Why? Because if some percentage of the population didn't believe in all kinds of Bad Science, there'd be no reason for me to be writing these columns. Let's imagine, optimistically, that it's only 1% of the TOP readership. Usually it's a lot higher. TOP has a readership measured in the five figures. You can do the math.

Should you feel compelled to write a fervent rebuttal to my branding your pet belief as Bad Science, Mike will happily publish your comment...so long as it is reasonably concise, not merely a political or religious rant (verboten topics, sez our Ed), and does not personally attack or insult anyone, including moi (no ad hominem). I will likely not respond to your rebuttal. That doesn't mean I will agree with you, only that I will not go out of my way to prove you wrong nor read some website that you're convinced will prove you're right. It's not my job nor my interest to tackle every bit of nonsense or bad information in the world.

Now, on with the show!

First, get as close to the source as possible. The more expertise your informant has, the closer they are to the research, the more likely they are to know what's good. If some columnist at Forbes or the Wall Street Journal says there is no scientific consensus on anthropogenic global warming and the editors of Science and Nature say there is, whom do you think really knows more about the science? C'mon, really. If Science and Nature are too dense for you to deal with, read Science News. It writes about science for the lay audience. It's not a primary source like Science and Nature, but it's a very, very close secondary source, written by people who actually have expertise.

Related to that, be suspicious when scientists step far outside their field of expertise and report unusual results. A lot of good scientists do Bad Science when they move into a discipline they're unfamiliar with, because it's a tricky thing getting experiments right. I'm a good enough experimental physicist; I wouldn't trust me to be able to run a good medical biology experiment.

Cold fusion's a good example. The discoverers knew so little physics that they didn't even know how to run some of their instruments correctly. They made some freshmen-level mistakes that were blatantly obvious to anyone who was minimally experienced with the equipment.

Next, consider the process. Did your informants submit papers for peer review? Did they get published in reputable and well-established journals or websites? If they're "taking it to the people" before doing any of that, that's a caution. Bad process usually means Bad Science. Again, see cold fusion.

Take a look at the scientific trend lines; these are really good indicators. All scientific theories and hypotheses, good or bad, start off with just a few adherents. Most of them, if they're not completely insane, rapidly acquire a fair percentage of followers because scientists collectively are open-minded and, not to put too fine a point on it, credulous. They'll be happy to provisionally believe in something until proven otherwise.

After achieving some modest popularity, though, Bad Science starts to lose supporters. More and more scientists become convinced that the data doesn't support the theory. Eventually you're left with only a small percentage of scientists believing in the Bad Science and the overwhelming majority abandoning it. Conversely, Good Science will eventually acquire a substantial supermajority. There'll remain a small percentage of naysayers into the indefinite future, which is why you look at the trend, not the exceptions.

Frequently, the trend can also be seen in the data. A bit of Bad Science will pop up because of an experiment or study that shows a large effect. When further experiments and studies fail to replicate that effect, or the effect comes and goes or becomes very small compared to the initial results and the noise in the experiment, those are all huge warning signs. Cold fusion failed both these kinds of trend lines.

Global warming denial fails the trend line. Deniers constantly trot out the hundreds of climate scientists who stand with them. What they don't mention is the tens of thousands of climate scientists who stand against them. Thirty years ago, that miniscule fraction might have been the start of a growing trend. Now, it isn't. It's the tail end of a declining minority. Anthropogenic climate change is settled science.

Watch out for the lone scientific "genius," crying out in the wilderness, whose work isn't appreciated or understood by the Establishment. That's a romantic notion promulgated in folklore and bad novels. In the real world, 99.9% of the time, the lone genius is a lone crackpot.

That's something you can look for in the discourse. The person who claims that they are asking questions that no one else dares to or that no one else has thought of, isn't. They are not smarter, cleverer, nor have a wider-ranging collective imagination than the thousands of other workers in that field. Any time you see someone saying in so many words, "I am so much more brilliant than all of them, because I have thought of this basic question that none of them have," you're seeing Bad Science. That one you can take to the bank. Don't bother reading any further in their screed; they will have nothing of value to tell you.

Another Bad Science sign is the "God of the Gaps" problem. This pops up in Creation Science (one of the grand old men of Bad Science) all the time. A classic tool of the Bad Scientist is to home in on some unresolved question and treat it as paramount, in essence saying, "Aha, science can explain A through Y, but they cannot explain Z. Therefore their science fails."

Nope, not even close. Science rarely has complete or perfect answers. There are always unresolved questions and gaps in our knowledge. Creation scientists love to point those out as proof that evolution isn't a viable theory. The problem is that the gaps keep getting smaller and smaller and of less and less importance. Remember what I said about trend lines? The god who was supposed to create entire species has been relegated to creating subtleties of molecular biochemistry and statistical selection processes. Most philosophers and theologians consider that a pretty piss-poor version of a god. Make a note: mixing science and religion? Always a bad idea for both of them.

This also shows up among the global warming deniers, who keep looking for the anomalous tree in a forest of coherent explanations. They have no trouble finding such; climatology and geophysics are really messy and incomplete sciences. Much Good Science is. Life is not always neat. Recognizing that does not make you brilliant.

•    •    •

Next time I'll pick up with the genuine nutters and crackpots, some of whom are genuinely dangerous.

©2013 by Ctein, all rights reserved

This is Ctein's 300th weekly column for TOP, which, according to the Editor's arithmetic, means he's been writing for us for .17333 years.

Links in this post may be to our affiliates; sales through affiliate links may benefit this site.

TOP's links!

(To see all the comments, click on the "Comments" link below.)
Featured Comments from:

Andrew Molitor: "t may be worthwhile to add a small compensating factor for money, where that quantity can be estimated. There's a lot of, if not Bad Science, at any rate Not Very Good Science which is done because there's grant money available. The relevant community may be hem and haw a bit on whether the science is any good, but by golly, that's a lot of money, so not very many will actually break ranks until the Science is shown to be really really Bad. That can take a while because, well, money. Cutting edge medicine seems to suffer from this a lot. It tends to sort itself out—eventually—but boy does it take a long time. Anyways. If you know about the money, and there's a great deal of it coming down on the Consensus side, you might want to squint at that side a little more closely."

Steve: "Try Conjectures and Refutations by Sir Karl Popper. Gets to the heart of this matter and is far from an impossible read.

Peter Horby: "I'm more than a little bemused by your characterization of Newton's theory of gravity as 'Bad Science.' To my layman's eye, Newtonian gravity is the poster child for Good Science. Every observation of the positions of celestial bodies for 250 years was explained by it, its predictive power was immense, and it still works perfectly well as the non-extreme limit of Einsteinian gravity. Good science is what Popper always said it was—a model with explanatory power which is falsifiable."

Ctein replies to Steve and Peter: I tend to be a Popperian (Popperite?) myself, but in fairness I should note it's not the only philosophy of science out there. There are other legitimate and useful ways to view the process. And, no, I can't write a column about that—I don't know this stuff well enough. Just mentioning this in case people are interested.

Newton's theory of gravity was Bad Science because it was fraud. At the time Newton proposed it, there were several competing theories of gravitation and the data was far too noisy to clearly favor one over the other. So Newton falsified his data—he published faked results that favored his theory!

This is the kind of fraud that is essentially impossible to detect. It's even likely to survive the Popper test because the theory is replicable.

Mendel, by the way, did essentially the same thing—he threw out breeding results that didn't fit the pattern he expected because he really didn't understand how statistics of small numbers work. In his case it probably wasn't deliberate fraud, he likely thought there was something genuinely wrong with those experiments. So it was more like malfeasance. Newton knew exactly what he was doing.

Throwing out data because you don't like the results when you can't see anything wrong with the experiment is poor process. It's one thing to eliminate it because you have a good reason for thinking it's wrong. (Just thinking, "Well, it doesn't fit with the rest of the data," is not a good reason.") That's why the superluminal neutrinos were an example of Good Science that was wrong. The researchers firmly believed the results were erroneous. The problem was that the experiment was so complex that they were unable to find the flaw in it. Which meant, under the precepts of good science, they had to release the results. They did so, clearly stating that they expected to be wrong and they needed help in finding out how they were wrong.

Of course, they hoped they were right, I mean, guaranteed Nobel Prize and all that. But they were pretty damn sure they weren't.

Steve Jacob: "I agree 100%. My degree was in chemistry, which gives me a smattering of physics, climatology and biochemistry. But I understand enough to know that my own opinion is worthless. I have not seen all the data, I am not qualified to judge the modelling involved, and I am certainly not up to date on all the prior research. But I can spot bogus nonsense or concocted statistics a mile away. They are normally reported in newspapers and popular news channels, usually out of context and using some strident quack as the only source. And I choose to accept (not on faith but as a working premise) that a large scientific consensus has a much better chance of being correct than a small number of vocal antagonists, especially if the question is one that has attracted a sufficient amount of attention, research and peer review. After all, just because I may not like the answer, or may even find the conclusion inconvenient or uncomfortable, none of that makes the slightest difference. My personal agenda is irrelevant. Clearly some people find that a hard thing to accept."

Mike Kukulski: "You both missed mentioning one 'bad science' trait and then used it yourself: 'xxx is settled science.' There is no such thing as 'settled science.'"

Ctein replies: Nope. Wrong. Settled science does not mean that there are no further questions to be asked. It means that the particular question under consideration has been settled. You can move on to details, refinements, exceptions, and extensions, but the starting question is no longer under debate. "Anthropogenic climate change" (and note that the whole phrase has to be considered, not the subset someone wants to argue about, nor additional adjectives that someone wants to tack on to set up their straw man) is settled science. The fundamental Theory of Evolution is settled science. The Atomic Theory is settled science. The Face On Mars (being ordinary geology and an illusion of lighting) is settled science.

Arguing those further is a waste of time and energy and resources and only gives a platform to the oddballs who are  otherwise at risk of losing their soapboxes. Keeping debate open on topics that no longer need to be debated is not a virtue, it is a vice. It is a waste of resources. It has no value.

Claiming "open-mindedness" is not a virtue when what you're being open-minded about is nonsense.

And, yes, engaging the God of the Gaps and the Conspiracies in the Closet as a way to maintain the veneer that there is still intelligent discussion to be resolved is nonsense.

Chris Lucianu: "Great column, generally. Minor issues:

1. Bad and Good Science need to be defined explicitly; especially if this Guide is aimed at the interested layperson.
2. The counterexamples (Newtonian gravitation; superluminal neutrinos) need to be explained for the sake of the layperson rather than merely mentioned in passing. I'm somewhat familiar with the matter, therefore I have an idea about what Ctein is trying to convey. Laypersons less conversant with the subject may just be passed by.

"It's tough to be a ruthless editor—which any good editor must be—with an author as august and as brilliant as Ctein. But I'm going to stick my neck out (fully expecting to have it chopped): I'd mark paragraphs #2 and #3 of the column with a squiggly line on the margin and send them respectfully back to the author with this note: 'Define. Exemplify. Clarify.'

"Objections: 'Make a note: mixing science and religion? Always a bad idea for both of them.' Nope. Just for religion. Whatever can be backed by scientific evidence ('Good Science' in Ctein's terms) is outside the scope of religions. Whatever is merely speculative is probably Bad Science (and bad philosophy, too). Whatever is just belief is just that: unsubstantiated belief.

"'Climatology and geophysics are really messy and incomplete sciences.' I'd argue that the subjects themselves are messy and complicated, not the science trying to untangle them. But so is reality. Our ideal is the simple, the elegant, the intuitive. Unraveling the complication and revealing the complexity is a huge step on the path of understanding."

Ctein replies: Unfortunately, in this case, I am the Ruthless Editor. I hacked and slashed like crazy. The two columns add up to 2500 words as it is. To give the subject proper treatment, I would've had to let them run twice that long but then they would've really fallen under the doctrine of 'longissimus, non legi' (thank you, Liza Furr!).

This is always a problem for online magazines; you simply can't write 3–5000 word articles and expect people to read them. So you leave out a whole bunch of interesting and germane stuff. Mike and I both do this all the time, 'cuz we haveta.

I figured I'd get to explaining the counterexamples in the comments and questions (which I are already have). I decided not to define Bad and Good Science, as people can pretty much figure out the definitions recursively, by reading my meta-rules.

As for mixing science and religion, that's a whole 'nother subject, but it does get mixed both ways annoyingly often and the results are almost invariably problematical. Attempting to establish your faith on the basis of science is building on pillars of sand; scientific paradigms are regularly overturned. Conversely, trying to make science follow your religious beliefs is not likely to go well, for obvious reasons.

Although it worked out very well for Keppler. Good Science inspired by exceedingly unlikely theology. There's always an exception.

Randal Jaffe: "Wonderful! As a scientist I try to make these points to my students. It is hard to explain to the general audience that science is not monolithic—sometimes the dissenting voice is correct. That is quite different from taking every dissenting voice as correct."

Bob Blakley: "Attempting to bend the subject back to photography, better instruments can support better science. Certainly TEPCO practiced bad science by using 100mSv Geiger counters to measure radiation they thought was in the range of 100mSv—because they were using the instrument at the edge of its range. When they switched to a better instrument last week they found that the real radiation intensity near Fukushima was about 1800mSv.

"Eadweard Muybridge improved science with a faster camera; he immediately settled the question of whether a horse's four hooves are simultaneously off the ground when it gallops using the new instrument. Edgerton took high-speed photography as a collector of scientific data to great heights. And Edison used it to invent motion pictures. Galileo built a better telescope (another imaging system!) to settle the question of whether there were bodies in the universe that did not revolve around (ahem) the Earth.

"At the end of the day, better data usually trumps cleverer theories, though the theories are what we like to argue about. And cameras can be very useful data collectors."

Jordan: "Conspiracy allegations are also a good way to quickly triage scientific claims—people who promote bad science tend to resort to conspiracy theories pretty quickly when attempting to get support for their work. You see this most clearly in the bio-medical field (alternative-medicine proponents alleging conspiracies that keep cancer 'cures' from coming to market, etc.) Another piece of the story is the role of the media in fuelling the confusion between good science and bad science. Lone wolves and underdogs make for great journalistic drama, but the tendency of reporters to go for 'both sides of the story' often ends up giving far too much weight to marginal work with little or no scientific support. And the problem gets magnified when the reporter has no scientific background (which is often the case)."

Stephen Scharf: "Regarding Ctein's comment 'be suspicious when scientists step far outside their field of expertise and report unusual results':

"While certainly Bad Science can result from this, history has proven that a lot of Good Science has also resulted from scientists stepping into another discipline; usually because that scientist is is not bound by conventional paradigmns and brings a fresh pair eyes and thinking to bear on scientific problems that have eluded the domain experts who are bound by 'in-the-box thinking.'

"For example, it was the experiments with the RII phage system that Seymour Benzer did that established that nucleic acids were the biological constituent that governed phenotypic inheritance. Benzer a trained as a physicist.

"Francis Crick was also trained in physics and did seminal work both in the discovery of the structure of DNA, conceived of the notion of transfer RNAs in protein translation and made contributions to neurobiology.

"The basis for antibody diversity eluded immunologists and cell biologists for decades until Philip Leder elucidated the genetic basis of antibody diversity. Leder was a molecular biologists, not an immunologist.

"I was sitting in a graduate protein biochemistry class years ago when the professor said that only 'protein-based' enzymes could demonstrate biochemical catalytic activity. When I asked him why RNAs could not demonstrate catalytic activity, he laughed in my face and said, 'Absolutely not, only proteins can serve as catalysts in the form of enzymes.' In 1982 (six years after my question in that biochemistry class...), Thomas Cech discovered catalytic RNAs and won the Nobel Prize in 1989. Cech was not a biochemist, either.

"So, while it may be true a signficant percentage of the time that scientists crossing over to other disciplines often do Bad Science, it is also true that truly significant breakthroughs have occured when scientists from other disciplines have done just that.

"Having a been a professional scientist my entire life, my formal training was in bacteriology, but virtually all of my significant work  has been in  disciplines I was not formally trained in, ranging from molecular biology (I'm one of the scientists at Cetus Corporation that developed the polymerase chain reaction, or PCR) to human genetics to molecular diagnostics to DNA forensics.

"Across the breadth of that career, if there's one thing I've learned, it's that it's not what scientific domain you're trained in that's important, it's how you think as a scientist that's important."