Steve Ziliak emails a reaction to a discussion I participated in on the role of randomized field experiments in economics:
Dear Mark,
I enjoyed the discussion that you, Hal Varian, and others had in The Economist, on the role of randomized field experiments in economics.
I'm surprised that Varian, the Chief Economist of Google, has not considered the evidence against randomization.
Your point about the mystery of uncontrolled structural change is a good one. But one can push the arguments against today's experiments even further. Completely randomized field experiments were tried and rejected more than a century ago, as it was found that completely randomized trials are not as precise, efficient, or powerful (in the Type II error sense) when compared against balanced designs.
The new field experimentalists have not considered these facts, nor have they tested their theory of randomization against the balanced alternatives proposed by Student and confirmed by Bayesians, decision theorists, and others, including Sir Harold Jeffreys, FRS and Egon S. Pearson, CBE, FRS.
Steve Levitt and John List (2009) have gone further than others by ignoring a century of evidence and theory against randomized trials. I believe that readers of Economist's View will want to know about a working paper I published at CREATES, the Center for Research in Econometric Analysis of Time Series at the University of Aarhus.
My paper, "Field Experiments in Economics: Comment on an Article by Levitt and List" (CREATES No. 2011-25), reviews and comments on a history and philosophy of field experiments published by Steve Levitt and John List (2009, European Economic Review) and cited and used by many, from Banerjee and Duflo at J-PAL to Karlan and others at the World Bank.
Here is the Abstract from my paper:
In an article titled "Field Experiments in Economics: The Past, the Present, and the Future," Levitt and List (2009) make three important claims about the history, philosophy, and future of field experiments in economics. They claim that field experiments in economics began in the 1920s and 1930s, in agricultural work by Neyman and Fisher. Second, they claim that artificial randomization is the sine qua non of good experimental design; they claim that randomization is the only valid justification for use of Student‘s test of significance. Finally, they claim that the theory of the firm will be advanced by economists doing randomized controlled trials (RCTs) for private sector firms. Several areas of economics, for example the development economics of Banerjee and Duflo, have been influenced by the article, despite the absence of historical and methodological review. This comment seeks to fill that gap in the literature. Student has, it is found, priority over Fisher and Neyman; he compared balanced and random designs in the field—on crops from barley to timber—from 1905 to 1937. The power and efficiency of balanced over random designs - discovered by Student and confirmed by Pearson, Neyman, Jeffreys, and others adopting a decision-theoretic and/or Bayesian approach - is not mentioned by Levitt and List. Neglect of Student is especially regrettable, for he showed in his job as Head Brewer of Guinness that artificial randomization is neither necessary nor sufficient for improving efficiency, identifying causal relationships, or discovering economically significant differences. One way forward is to take a step backwards, from Fisher to Student.All the best,
Steve Ziliak
"
Comments