I Just Want to See the Raw Data!

Originally posted (by me) on LinkedIn, December 9, 2014.

This is pretty close to raw data. How informative is it?

This is pretty close to raw data. How informative is it?

“I just want to see the raw data! No interpretation, no massaging the numbers, just the raw data straight out of the instrument!”

I sympathized with my non-scientist friend. She felt frustrated after reading a series of news items that began with a promising discovery, followed by a series of caveats, followed by more news stories reporting that no one knew for sure what was going on, and several more years of research would be required to clarify the findings from the initial report.

She didn’t know whom to believe. Scientists presented what looked like clear and convincing evidence, only to be shouted down by political activists and religious leaders claiming, “That’s just your opinion!” and citing past scientific studies proven biased, fraudulent, or just plain wrong.

The problem is, raw data points don’t tell you much of anything. Even an experienced scientist needs some kind of interpretation to convert the numbers into knowledge. What question was being asked and how did the person go about trying to find an answer? What did the instrument measure directly, and what assumptions were used to make indirect observations? What good does it do you to know that an unknown sample produces six times the voltage response at 7.3 minutes elution time than it did at 6.9 minutes if that’s all you know?

What were the conditions of the experiment? The same instrument can generate high- and low-resolution data, depending on how it is set up. Each type of data is useful for some purposes, but not others. Attachments, filters, thermostats — each of these can be added, adjusted, or calibrated to increase the sensitivity toward some observations, but they can also obscure other observations. The star you’re looking at might shine most brightly at the wavelengths you’re filtering out to cut interference from the sodium-vapor street lamps nearby.

How closely do the observations mimic processes in the real world? Lab-scale syntheses often fail to predict the results of a full-scale industrial production run. Data taken in the field under real-life conditions can be biased or inaccurate, because the act of observing has altered the behavior of thing you’re observing.

The progress of science itself can prove previous science wrong or expose the limitations of previous theories. Some 19th-century physicists thought that physics would shortly become a closed field of inquiry. All the questions had been answered, they thought. Only a few small loose ends needed to be wrapped up, and then the books could be closed. One of those loose ends turned out to be quantum theory, which underlies the technologies behind the automatic doors at the grocery store and flash drives that let you store hundreds of tunes in a device you can carry in your pocket.

Scientists do share raw data among themselves, especially when they are seeking alternate interpretations or reusing data for another purpose. The studies they publish in the journals contain interpreted data, along with information on how the data were obtained and what assumptions and processing steps were used. Other scientists with experience in the strengths and limitations of various instruments and methods review each others’ studies and identify gaps or alternate interpretations.

Some tests have been repeated so often, with such consistent results, that the interpretative steps can be programmed into the instruments themselves. You see this type of lab analysis on television shows where the forensic lab tech puts a paint chip from a crime scene into an instrument, and immediately sees that it could only have come from a 2009 Fiat. This data is not raw — the interpretation has merely been automated.

A wide spectrum of knowledge spans the territory between “That’s just your opinion!” and “There’s so much evidence here that I would stake my life on this.” The difference lies not in seeking some pure spring of unsullied data, but in knowing what questions were asked, how they were answered, and how the answers fit in with everything else. It requires seeing things happen the same way over and over and trusting things to happen that way again under the same conditions. It also requires a willingness to change your thinking if new information puts established knowledge into a new and broader context.

Update: If you would just listen to the facts…

Janet Raloff, a friend of mine and a senior editor at Science News, has weighed in on the topic of how scientists can communicate effectively with the public on divisive issues such as climate change. In her May 29 online article, Climate skepticism not rooted in science illiteracy, she reports on an interview with Dan Kahan of Yale Law School, one of the authors of the Nature Climate Change article I cited in my previous posting. Raloff explores several approaches to interacting with a polarized public in her May 30 follow-up article, Depolarizing climate science.

What comes through most strongly in these articles is that divisions occur when scientific findings have a direct relevance to things that affect us every day, or findings that could make significant changes in our everyday lives. As far as I know, The Heartland Institute has not posted any billboards comparing people who believe in black holes or the law of gravity with Unabomber Ted Kaczynski. We accept these things because we can do so without making any changes in our daily routines.

Raloff cites political scientist Arthur Lupia of the University of Michigan in Ann Arbor, who states that some of the most science-literate critics will listen to experts only to generate compelling counterarguments. What’s stopping climate scientists from doing the same thing — listening to the most rational, articulate deniers and opponents of climate change legislation in order to scope out what matters most to climate change deniers and formulating compelling counterarguments of their own?

Disruptive innovation is another possible approach — one that sidesteps the battleground altogether by providing people with alternatives that they adopt readily because they see a clear benefit to the new technologies. After all, no one waged a political war against paper phone books, wristwatches, or the Sony Walkman, and yet those things have almost completely fallen out of use by Americans under the age of 30, in favor of smartphones, smartphones, and smartphones. The main problem with this approach is that disruptive innovations are notoriously unpredictable. We cannot guarantee the timely arrival and widespread adoption of any particular form of alternative energy or resource-conserving capabilities.

So for now, we are stuck with learning to talk to each other. Which is something that we needed to do anyway.

If you would just listen to the facts…

Julia Duin’s article in the May 29 Washington Post (Serpent-handling pastor profiled earlier in Washington Post dies from rattlesnake bite) bothered me a lot, for reasons that I am still exploring. Duin is a photojournalist who has on several occasions reported on a small Pentecostal Christian group in rural West Virginia. This group demonstrates their religious faith by handling poisonous snakes during their church services. They cite Mark 16:17-18: “And these signs will follow those who believe: …they will take up serpents; and if they drink anything deadly, it will by no means hurt them…”. Mack Wolford, the pastor of this church, died shortly after a rattlesnake bit him during a “homecoming” celebration over the Memorial Day weekend. His father, the previous pastor, died from a poisonous snakebite in 1983.

This may seem like an odd topic for a science blog, but bear with me a bit. It’s that very unfamiliarity of this whole practice (for most people who are likely to be reading this blog, anyway) that illustrates a problem that scientists have in communicating their research to the public.

Duin struggled over whether she should have called for medical help, which Wolford refused for several hours after he was bitten. This is a perennial issue for photojournalists, who must weigh their responsibility to serve as reliable witnesses to event against their responsibility to offer aid to a fellow human in danger. She wrote about that aspect much better than I could have, so I refer you to her article for that discussion. Basically, these people weren’t ignorant about calling for medical help, they just chose not to.

I sorted through a whole pile of mental conflicts as I read Duin’s article. How far should we take religious tolerance? They weren’t flying airplanes into buildings, after all, but they were trying to win over new recruits to the practice, even in states where it is illegal. These people practice their religion in remote rural areas, so there is little danger of their snakes escaping to terrorize hundreds of office workers taking their lunch break in Farragut Square. On the other hand, we do raise an uproar over other practices such as female genital mutilation that have remained “out of sight, out of mind” until only recently. We’re all connected by our common humanity, so we can’t really say that what “they” do is all right as long as they are only hurting themselves — their pain is ours as well.

What about that story where Satan tempted Jesus to prove his divinity by jumping off the roof of the temple and letting the angels rescue him? Satan quoted a piece of scripture that seemed to support this, much as the snake handlers do. Jesus answered, “It is also written: ‘Do not put the Lord your God to the test.'” In other words, you have protection when you need it, but it’s not a free pass for taking foolish risks. The fact that your insurance company will pay for your repairs if you get into an auto accident isn’t an encouragement for you to drive down the freeway at 125 miles an hour. Shouldn’t someone point this out to the snake handlers?

This last question is where I make the mental connection to science communications. The snake handlers don’t need anyone to point out that snake bites are dangerous and can kill you. They know it already, they’ve seen it up close. They don’t need anyone to teach them scripture — they probably know it better than most people. They are not suffering from a dearth of information. As one respondent to Duin’s article pointed out, their main concern is not whether they die, but how they die. In their value system, dying while demonstrating their faith is far preferable to living in ambiguity.

You see the same sort of thing with climate change deniers. Their problem is not ignorance. Very few of these people would change their stance after attending a scientific seminar on the topic. As an increasing body of peer-reviewed studies shows, people form their views based not only on the information they have, but also on their vested interests (financial and otherwise), their desire to fit in with their families and communities, the degree to which they trust the source of information, and how this information fits in with their values and established world views.

A recent  study in Nature Climate Change illustrates the futility of merely heaping more information on people in an attempt to change their minds. The authors of the Nature article conclude:
It does not follow, however, that nothing can be done to promote constructive and informed public deliberations. As citizens understandably tend to conform their beliefs about societal risk to beliefs that predominate among their peers, communicators should endeavor to create a deliberative climate in which accepting the best available science does not threaten any group’s values. Effective strategies include use of culturally diverse communicators, whose affinity with different communities enhances their credibility, and information-framing techniques that invest policy solutions with resonances congenial to diverse groups. Perfecting such techniques through a new science of science communication is a public good of singular importance.

Even this, however, is only a partial solution, focusing as it does on coming up with more acceptable messengers in what is essentially a one-way flow of information. I would go one step further: listen first. Find out what other people think. Ask them why they think the way they do. Point out the places where you can’t understand their reasoning, or where their actions fly in the face of something you think is obvious or morally imperative, but don’t do it to sell them on your point of view. Do it to point out the areas where you need to learn more about how they think and believe.

Listening breaks the pointless cycle of attack and defense and establishes a starting point for a constructive engagement. You might never reach agreement, but in the act of listening and respecting another point of view, you make it more likely that the other person will listen to you in return. In establishing a dialogue, you make it more likely that you will identify commonalities that can serve as the beginning of constructive action. You establish yourself as a trustworthy source of information, making it more likely that the other person will listen to you in the future. And who knows, you might even learn something new.