"You keep using that word. I do not think it means what you think it means."
Erich Huang and Brian Bot of Sage Bionetworks pose the problem of reproducible science this way: “[S]cientific claims are artificially abstracted from the science itself. This is a real issue in the data-intensive sciences." I think that’s true, but I would go further. I don’t think we really know what our "scientific claims" actually are, and we may have a tentative grasp of the "science," too.
That is, we really don’t know results, and that is a large part of our confusion in data-intensive biomedical research.
I have to admit that I still find the terse declaratives of scientific publication titles a little jarring. They declare the point of the whole publication, of course, or at least the most important and telling “result” — something along the line of, say, “SATB1 reprogrammes gene expression to promote breast tumour growth and metastasis” [Nature]. That article inspired Elizabeth Iorns to do some follow up research, but she ran into trouble — she couldn’t reproduce the original work. As Carl Zimmer reports in Slate and as appeared in Nature, the experience led Iorns to set up a for-profit company focused on reproducing research, since so much research isn’t reproducible and incentives to clean out the irreproducible lack in the way we’ve structured scientific endeavor.
In academic science, we’re apparently acting blindly, though the venture capital world (where “results” matter differently) notes with alarm “that at least 50% of the studies published even in top tier academic journals … can’t be repeated” [LifeSci VC].
Studies, results, scientific claims — these words seem to have a certain fuzzy equivalence (something akin to the words reproducible and repeatable which are actually distinct in their specific meanings). It is possible to redo a “study” — defined as an ordered set of tasks and activities believed to explore certain phenomena — without being able to get the same result. The result then isn’t reproducible, at least through the method defined in the study. But too frequently we leap to the result, focusing on it as the essential problem of reproducibility, when actually a result may be a better indicator of the adequacy of the method or of the “study.”
Essentially, “results” are tied to methods much more tightly than our current scientific publication modes let on. Look for the smallest font size in a scientific journal and you find the “methods section.” There’s a reason for the small typeface, and it seems to me the literal diminishment of method is a clear problem for science.
Really, and perhaps unknown to them right now, Huang and Bot are moving toward a reordering of fundamental priorities, a statement of what science is all about. They probably believe that they are reconfiguring scientific publication in large part to retool science for reproducibility. But something bigger is going on here: scientific claims and results are being fundamentally redefined. The clearScience initiative [http://blog.clearscience.io/] appears to me to recognize that the conventional (and slavishly formulaic) distinction between what have been called results and the way you get to those results is … false and contrived. I would say that distinction comes in part from the form of scientific publication — a rhetorical choice, in some sense, that enshrines factoidicity while tamping down the importance of argument that, after all, is the way you bring your audience to the point of the piece.
I am among the woolly headed thinkers in the genome sciences. Fairly well trained in the humanities and taught to savor words, I have to admit that scientific publication is dry as dust, more a desert of verbiage than dry light of reason. The clearScience initiative may not turn scientific publications into prose masterpieces, but I do think it has promise to bring together method, argument, data, and result in a satisfyingly rich fashion expressing scientific endeavor more completely and reliably.
At any rate, I hope the time comes when a scientific “result” no longer fits well on a bumper sticker. A real “result” means something more and strives for a greater unity of scientific narrative (however expressed) and of What Happens Then, Near The End.