The question of what to do with nullresults—when researchers fail to see an effect thatshould be detectable—has long been hotly debatedamong those conducting medical trials, where theresults can have a big impact on lives and corporatebottom lines. More recently, the debate has spread tothe social and behavioral sciences, which also havethe potential to sway public and social policy.There were little hard data, however, on how often orwhy null results were squelched. “Yes, it’s true thatnull results are not as exciting,” political scientistGary King of Harvard University says. “But I suspectanother reason they are rarely published is that thereare many, many ways to produce null results bymessing up. So they are much harder to interpret.”In a recent study, Stanford political economistNeil Malhotra and two of his graduate studentsexamined every study since 2002 that was funded bya competitive grants program called TESS(Time-sharing Experiments for the Social Sciences).TESS allows scientists to order up Internet-basedsurveys of a representative sample of US adults to testa particular hypothesis (for example, whether voterstend to favor legislators who boast of
bringing federaldollars to their districts over those who tout a focuson policy matters).Malhotra’s team tracked down working papersfrom most of the experiments that weren’t published,and for the rest asked grantees what had happened totheir results. In their e-mailed responses, somescientists cited deeper problems with a study or morepressing matters—but many also believed thejournals just wouldn’t be interested. “Theunfortunate reality of the publishing world [is] thatnull effects do not tell a clear story,” said onescientist. Said another, “Never published, definitelydisappointed to not see any major effects.”Their answers suggest to Malhotra that rescuingfindings from the file drawer will require a shift inexpectations. “What needs to change is theculture—the author’s belief about what will happen ifthe research is written up,” he says.Not unexpectedly, the statistical strength of thefindings made a huge difference in whether theywere ever published. Overall, 42% of the experiments produced statistically significant results. Of those,62% were ultimately published, compared with 21%of the null results. However, the Stanford team wassurprised that researchers didn’t even write up65% of the experiments that yielded a null finding.Scientists not involved in the study praise its“clever” design. “It’s a very important paper” that“starts to put numbers on things we want tounderstand,” says economist Edward Miguel of theUniversity of California, Berkeley.He and others note that the bias against nullstudies can waste time and money when researchersdevise new studies replicating strategies alreadyfound to be ineffective. Worse, if researchers publishsignificant results from similar experiments in thefuture, they could look stronger than they shouldbecause the earlier null studies are ignored. Evenmore troubling to Malhotra was the fact that twoscientists whose initial studies “didn’t work out”went on to publish results based on a smaller sample.“The non-TESS version of the same study, in whichwe used a student sample, did yield fruit,” noted oneinvestigator.A registry for data generated by all experimentswould address these problems, the authors argue.They say it should also include a “preanalysis” plan,that is, a detailed description of what the scientisthopes to achieve and how the data will be analyzed.Such plans would help deter researchers fromtweaking their analyses after the data are collected in search of more publishable results.
