Tuesday, July 1, 2014

The problem with data

The progress of science requires inspiration.  Some researchers find this only from data.  "Show me the evidence", they say.  Many peer-reviewed publications in science, however, have no data.  They involve a different kind of inspiration:  proposals for original ideas or new hypothesis development.  These are found within the 'Forum', 'Perspectives', 'Opinion' and 'Commentary' sections of many journals, and in some journals devoted entirely to new ideas and commentary.

I have always been particularly drawn to the honesty and beauty in this creative brand of enquiry.  And so I am puzzled to see it often dismissed out of hand with pejorative labeling, like ‘hand-waving’ and ‘just-so-stories’.  Many — especially among the elites and self-appointed guardians of established theory  would have us believe that only ‘evidence-based’ practice and products can be taken seriously as legitimate sources of inspiration and discovery in science. 

This is plainly arrogant and wrongheaded.  The scientific method means doing whatever is necessary to get good answers to questions worth asking.  And so data collection that is not guided by interesting, novel, and important ideas is usually boring at best.  At worst, it is a waste of research grant funds.  

But published data are plagued with an even more serious problem:  we never know how much to trust it.  A few minutes of Google searching under the terms "research bias", "scientific misconduct", and "publication bias" shows that the follies of faith in published data have come sharply and painfully into the public spotlight in recent years.  Data are also subject to measurement and recording errors, and are often not replicated, in some cases because they can't be, e.g. when collection materials or settings are not repeatable (which is often the case in my own field of research in ecology and evolution).  

Well-established researchers have always been privately aware of these troubling limitations of published data, and many have had direct knowledge of the scale of the problem — usually while showing a brave face for the public, intimating that the scientific establishment is virtuous, and it’s integrity (save for a few ‘isolated cases’), rock solid.  

This would be amusing if it weren't so egregious and pathetic. 

Good data will of course always contribute importantly to the progress of science, and in a perfect world, ‘evidence-based’ would be the gold standard.  But unfortunately, proclamations of evidence are rarely free of suspicion.  Both researchers and the general public are well-advised to be wary of the potential for false confidence, questionable inspiration, and misguided recommendations.  

Remedies for the data problem remain an important challenge for researchers.  In the meantime, however, a rich and reliable source of inspiration  — thankfully  remains strong in supporting the progress of science:  published ideas and stories.  Write them, publish them, read them, challenge them, revise them, and be inspired!