
1 
Paper

Agnostic Notes on Regression Adjustments to Experimental Data:
Reexamining Freedman's Critique
Lin, Winston

Uploaded 
09022011

Keywords 
Covariate adjustment Randomization inference Neyman's repeated sampling approach Sandwich estimator Social experiments

Abstract 
Freedman [Adv. in Appl. Math. 40 (2008a) 180–193; Ann. Appl. Stat. (2008b) 2 176–196] critiqued OLS regression adjustment of estimated treatment effects in randomized experiments, using Neyman’s model for randomization inference. This paper argues that in sufficiently large samples, the statistical problems he raised are either minor or easily fixed. OLS adjustment improves or does not hurt asymptotic precision when the regression includes a full set of treatmentcovariate interactions. Asymptotically valid confidence intervals can be constructed with the HuberWhite sandwich standard error estimator. Even the traditional OLS adjustment has benign largesample properties when subjects are randomly assigned to two groups of equal size. The strongest reasons to support Freedman’s preference for unadjusted estimates are transparency and the dangers of specification search. 

2 
Paper

Reasoning about Interference Between Units}
Bowers, Jake
Fredrickson, Mark
Panagopoulos, Costas

Uploaded 
07132012

Keywords 
interference randomization inference SUTVA randomized experiments Fisher's sharp null hypothesis causal inference

Abstract 
If an experimental treatment is experienced by both treated and control group units, tests of hypotheses about causal effects may be difficult to conceptualize let alone execute. In this paper, we show how counterfactual causal models may be written and tested when theories suggest spillover or other networkbased interference among experimental units. We show that the ``no interference'' assumption need not constrain scholars who have interesting questions about interference. We offer researchers the ability to model theories about how treatment given to some units may come to influence outcomes for other units. We further show how to test hypotheses about these causal effects, and we provide tools to enable researchers to assess the operating characteristics of their tests given their own models, designs, test statistics, and data. The conceptual and methodological framework we develop here is particularly applicable to social networks, but may be usefully deployed whenever a researcher wonders about interference between units. Interference between units need not be an untestable assumption; instead, interference is an opportunity to ask meaningful questions about theoretically interesting phenomena. 

3 
Paper

Attributing Effects to A Cluster Randomized GetOutTheVote Campaign: An Application of Randomization Inference Using Full Matching
Bowers, Jake
Hansen, Ben

Uploaded 
07182005

Keywords 
causal inference randomization inference attributable effects full matching instrumental variables missing data field experiments clustering

Abstract 
Statistical analysis requires a probability model: commonly, a model
for the dependence of outcomes $Y$ on confounders $X$ and a
potentially causal variable $Z$. When the goal of the analysis is to
infer $Z$'s effects on $Y$, this requirement introduces an element
of circularity: in order to decide how $Z$ affects $Y$, the analyst
first determines, speculatively, the manner of $Y$'s dependence on
$Z$ and other variables. This paper takes a statistical perspective
that avoids such circles, permitting analysis of $Z$'s effects on
$Y$ even as the statistician remains entirely agnostic about the
conditional distribution of $Y$ given $X$ and $Z$, or perhaps even
denies that such a distribution exists. Our assumptions instead
pertain to the conditional distribution $Z vert X$, and the role of
speculation in settling them is reduced by the existence of random
assignment of $Z$ in a field experiment as well as by
poststratification, testing for overt bias before accepting a
poststratification, and optimal full matching. Such beginnings pave
the way for ``randomization inference'', an approach which, despite
a long history in the analysis of designed experiments, is
relatively new to political science and to other fields in which
experimental data are rarely available.
The approach applies to both experiments and observational studies.
We illustrate this by applying it to analyze A. Gerber and
D. Green's New Haven Vote 98 campaign. Conceived as both a
getoutthevote campaign and a field experiment in political
participation, the study assigned households to treatment and
desired to estimate the effect of treatment on the individuals
nested within the households. We estimate the number of voters who
would not have voted had the campaign not prompted them to  that
is, the total number of votes attributable to the interventions of
the campaigners  while taking into account the nonindependence
of observations within households, nonrandom compliance, and
missing responses. Both our statistical inferences about these
attributable effects and the stratification and matching that
precede them rely on quite recent developments from statistics; our
matching, in particular, has novel features of potentially wide
applicability. Our broad findings resemble those of the original
analysis by citet{gerbergreen00}. 

4 
Paper

Making Inferences from 2x2 Tables: The Inadequacy of the Fisher Exact\r\nTest for Observational Data and a Principled Bayesian Alternative
Sekhon, Jasjeet

Uploaded 
08172005

Keywords 
Fisher exact test randomization inference permutation
tests Bayesian tests difference of proportions observational data

Abstract 
The Fisher exact test is the dominant method of making inferences from
2x2 tables where the number of observations is small. Although the
Fisher test and approximations to it are used in a large number of
studies, these tests rest on a data generating process which is
inappropriate for most applications for which they are used. The
canonical Fisher test assumes that both of the margins in a 2x2 table
are fixed by constructioni.e., both the treatment and outcome
margins are fixed a priori. If the data were generated by an
alternative process, such as binomial, negative binomial or Poisson
binomial sampling, the Fisher exact test and approximations to it do
not have correct coverage. A Bayesian method is offered which has
correct coverage, is powerful, is consistent with a binomial process
and can be extended easily to other distributions. A prominent 2x2
table which has been used in the literature by Geddes (1990) and
Sekhon (2004) to explore the relationship between foreign threat and
social revolution (Skocpol, 1979) is reanalyzed. The Bayesian method
finds a significant relationship even though the Fisher and related
tests do not. A Monte Carlo sampling experiment is provided which
shows that the Bayesian method dominates the usual alternatives in
terms of both test coverage and power when the data are generated by a
binomial process. 

5 
Paper

Variance Identification and Efficiency Analysis in Randomized Experiments under the MatchedPair Design
Imai, Kosuke

Uploaded 
07172007

Keywords 
Average Treatment Effect Causal Inference Experimental Design Matched Samples Paired Comparison Randomization Inference.

Abstract 
In his landmark article, Neyman (1923) introduced randomizationbased inference in analyzing experiments under the completely randomized design. Under this framework, Neyman considered the statistical estimation of the sample average treatment effect and derived the variance of the standard estimator using the treatment assignment mechanism as the sole basis of inference. In this paper, I extend Neyman's analysis to randomized experiments under the matchedpair design where experimental units are paired based on their pretreatment characteristics and the randomization of treatment is subsequently conducted within each matched pair. I study the variance identification for the standard estimator of average treatment effects and analyze the relative efficiency of the matchedpair design over the completely randomized design. I also show how to empirically evaluate the relative efficiency of the two designs using experimental data obtained under the matchedpair design. My randomizationbased analysis clarifies some of the important questions raised in the literature and identifies a hiden and yet implausible assumption that is made for the efficiency analysis in a widely used textbook. Finally, the analytical results are illustrated with numerical and empirical examples. 

6 
Paper

A General Method for Detecting Interference Between Units in Randomized Experiments
Aronow, Peter

Uploaded 
08172010

Keywords 
Rubin Causal Model SUTVA Permutation test Causal inference Randomization inference

Abstract 
Interference between units may pose a threat to unbiased causal inference in randomized controlled experiments. Although the assumption of no interference is essential for causal inference, few options are available for testing this assumption. This paper presents the first reliable ex post method for detecting interference between units in randomized experiments. Naive estimators of interference that attempt to exploit the proximity of units may be biased because simple randomization of units into treatment does not imply simple randomization of proximity to treated units. However, through a randomizationbased approach, the confounding associated with these naive estimators may be circumvented entirely. With a test statistic of the analyst's choice, a conditional randomization test allows for the calculation of the exact significance of the causal dependence of outcomes on the treatment status of other units. The efficacy and robustness of the method is demonstrated through simulation studies and, using this method, interference between units is detected in a field experiment designed to assess the effect of mailings on voter turnout. 

