Essays on Causal Inference in Randomized Experiments


Book Description

This dissertation explores methodological topics in the analysis of randomized experiments, with a focus on weakening the assumptions of conventional models. Chapter 1 gives an overview of the dissertation, emphasizing connections with other areas of statistics (such as survey sampling) and other fields (such as econometrics and psychometrics). Chapter 2 reexamines Freedman's critique of ordinary least squares regression adjustment in randomized experiments. Using Neyman's model for randomization inference, Freedman argued that adjustment can lead to worsened asymptotic precision, invalid measures of precision, and small-sample bias. This chapter shows that in sufficiently large samples, those problems are minor or easily fixed. OLS adjustment cannot hurt asymptotic precision when a full set of treatment-covariate interactions is included. Asymptotically valid confidence intervals can be constructed with the Huber-White sandwich standard error estimator. Checks on the asymptotic approximations are illustrated with data from a randomized evaluation of strategies to improve college students' achievement. The strongest reasons to support Freedman's preference for unadjusted estimates are transparency and the dangers of specification search. Chapter 3 extends the discussion and analysis of the small-sample bias of OLS adjustment. The leading term in the bias of adjustment for multiple covariates is derived and can be estimated empirically, as was done in Chapter 2 for the single-covariate case. Possible implications for choosing a regression specification are discussed. Chapter 4 explores and modifies an approach suggested by Rosenbaum for analysis of treatment effects when the outcome is censored by death. The chapter is motivated by a randomized trial that studied the effects of an intensive care unit staffing intervention on length of stay in the ICU. The proposed approach estimates effects on the distribution of a composite outcome measure based on ICU mortality and survivors' length of stay, addressing concerns about selection bias by comparing the entire treatment group with the entire control group. Strengths and weaknesses of possible primary significance tests (including the Wilcoxon-Mann-Whitney rank sum test and a heteroskedasticity-robust variant due to Brunner and Munzel) are discussed and illustrated.




Essays in Causal Inference and Experimental Design


Book Description

This dissertation is comprised of two chapters the first of which is concerned with how to address a prevalent issue when analyzing data from randomized experiment with human participants, and the second of which investigates aspects of experimental design. In Chapter 1 I study the problem of identifying the causal effect of an experimental treatment when the experiment suffers from non-compliance. In particular, I consider the identifying power of information collected from non-complying participants after the completion of the treatment phase. Follow up surveys often ask study participants why they chose not to accept an offer of treatment despite being assigned to it, and answers to such questions offer insights into the decision process by which agents choose to comply with their assigned treatment status. I propose a model that rationalizes an agent's compliance decision. Based on this model, I characterize the set of values for the average treatment effect that are conformable with data observed in a randomized trial. This model and the implied set of identified values for the average treatment effect rely on the availability of follow up surveys that ask agents why they chose to not comply. This underscores the importance of following up with non-complying agents since the model often leads to substantially tighter identified sets for the average treatment effect than what is possible without this information. I apply the proposed model to data from the Job Training Partnership Act Study to estimate identified sets for the average treatment effect for a number of employment outcomes. Chapter 2 considers the problem of where to introduce a new policy. Field experiments can inform where to implement policies, but in practice there may be hundreds of candidate sites being considered and concerns about external validity. We develop a quasi-Bayesian approach to selecting a small number of sites in which to run new experiments to inform policy choices across a wide set of locations. The approach develops a prior specification for the joint distribution of site-level average treatment effects based on a microeconometric structural model, while also allowing for heterogeneity across sites that is not captured by the model. We apply the approach to choices over rolling out a mobile money innovation in 41 migration corridors in Bangladesh, 60 in Pakistan, and 740 in India. Learning is optimized when the experiments take place in locations informative for the largest number of sites where suitability of the policy is a priori uncertain. Learning and expected welfare improve relative to experimentation based on plausible rules of thumb.




Essays in Causal Inference


Book Description

In observational studies, identifying assumptions may fail, often quietly and without notice, leading to biased causal estimates. Although less of a concern in randomized trials where treatment is assigned at random, bias may still enter the equation through other means. This dissertation has three parts, each developing new methods to address a particular pattern or source of bias in the setting being studied. In the first part, we extend the conventional sensitivity analysis methods for observational studies to better address patterns of heterogeneous confounding in matched-pair designs. We illustrate our method with two sibling studies on the impact of schooling on earnings, where the presence of unmeasured, heterogeneous ability bias is of material concern. The second part develops a modified difference-in-difference design for comparative interrupted time series studies. The method permits partial identification of causal effects when the parallel trends assumption is violated by an interaction between group and history. The method is applied to a study of the repeal of Missouri's permit-to-purchase handgun law and its effect on firearm homicide rates. In the final part, we present a study design to identify vaccine efficacy in randomized control trials when there is no gold standard case definition. Our approach augments a two-arm randomized trial with natural variation of a genetic trait to produce a factorial experiment. The method is motivated by the inexact case definition of clinical malaria.




Essays in Causal Inference and Public Policy


Book Description

This dissertation addresses statistical methods for understanding treatment effect variation in randomized experiments, both in terms of variation across pre-treatment covariates and variation across post-randomization intermediate outcomes. These methods are then applied to data from the National Head Start Impact Study (HSIS), a large-scale randomized evaluation of the Federally funded preschool program, which has become an important part of the policy debate in early childhood education.




Essays in Causal Inference


Book Description

This dissertation explores the estimation of causal effects in settings with non-standard data. In the first chapter, the treatments are not directly assigned to outcome units but instead occur in the same geographic space. In the second chapter, responses to hypothetical questions describing both the treated and control state are used to learn about the effects of a treatment on real behavior (outcomes). In the third chapter, treatments are assigned according to a randomized experiment but outcomes are heavy-tailed, such that semiparametric approaches are useful to improve efficiency and robustness. The first chapter considers settings where the treatments causing the effects of interest are not directly associated with specific units for which we measure outcomes, but rather occur in the same geographic space. Many events and policies (treatments), such as opening of businesses, building of hospitals, and sources of pollution, occur at specific spatial locations, with researchers interested in their effects on nearby individuals or businesses (outcome units). However, the existing treatment effects literature primarily considers treatments that could experimentally be assigned directly at the level of the outcome units, potentially with spillover effects. I approach the spatial treatment setting from a similar experimental perspective: What ideal experiment would we design to estimate the causal effects of spatial treatments? This perspective motivates a comparison between individuals near realized treatment locations and individuals near counterfactual (unrealized) candidate locations, which is distinct from current empirical practice. I derive standard errors based on this design-based perspective that are straightforward to compute irrespective of spatial correlations in outcomes. Furthermore, I propose machine learning methods to find counterfactual candidate locations and show how to apply the proposed methods on observational data. I study the causal effects of grocery stores on foot traffic to nearby businesses during COVID-19 shelter-in-place policies. I find a substantial positive effect at a very short distance. Correctly accounting for possible effect "interference" between grocery stores located close to one another is of first order importance when calculating standard errors in this application. The second chapter is co-authored with B. Douglas Bernheim, Daniel Björkegren, and Jeffrey Naecker. We explore methods for inferring the causal effects of treatments on choices by combining data on real choices with hypothetical evaluations. We propose a class of estimators, identify conditions under which they yield consistent estimates, and derive their asymptotic distributions. The approach is applicable in settings where standard methods cannot be used (e.g., due to the absence of helpful instruments, or because the treatment has not been implemented). It can recover heterogeneous treatment effects more comprehensively, and can improve precision. We provide proof of concept using data generated in a laboratory experiment and through a field application. The final chapter is co-authored with Susan Athey, Peter J. Bickel, Aiyou Chen, and Guido W. Imbens. We develop new semiparametric methods for estimating treatment effects. We focus on a setting where the outcome distributions may be heavy-tailed, where treatment effects are small, where sample sizes are large and where assignment is completely random. This setting is of particular interest in recent experimentation in tech companies. We propose using parametric models for the treatment effects, as opposed to parametric models for the full outcome distributions. This leads to semiparametric models for the outcome distributions. We derive the semiparametric efficiency bound for this setting, and propose efficient estimators. In the case with a constant treatment effect one of the proposed estimators has an interesting interpretation as a weighted average of quantile treatment effects, with the weights proportional to (minus) the second derivative of the log of the density of the potential outcomes. Our analysis also results in an extension of Huber's model and trimmed mean to include asymmetry and a simplified condition on linear combinations of order statistics, which may be of independent interest.




Field Experiments and Their Critics


Book Description

In recent years, social scientists have engaged in a deep debate over the methods appropriate to their research. Their long reliance on passive observational collection of information has been challenged by proponents of experimental methods designed to precisely infer causal effects through active intervention in the social world. Some scholars claim that field experiments represent a new gold standard and the best way forward, while others insist that these methods carry inherent inconsistencies, limitations, or ethical dilemmas that observational approaches do not. This unique collection of essays by the most influential figures on every side of this debate reveals its most important stakes and will provide useful guidance to students and scholars in many disciplines.




Three Essays on Causal Inference


Book Description

This thesis describes three research projects in causal inference, all related to the problem of contrasting the average counterfactual outcomes on two sides of a binary decision. In the first project, we discuss estimation of the average causal effect in a randomized control trial. Here, we find that statisticians find themselves in a kind of statistical paradise: a simple model-based procedure delivers correct confidence intervals even if the experimental participants are not randomly sampled and mis-specified models are used. In the second project, we consider the problem of testing for a treatment effect using observational data with no hidden confounders. Conceptually, this is no different from a rather complicated RCT, and one might expect that a return to statistical paradise is possible. Unfortunately, this is not the case: we show that even intuitively reasonable uses of correct models may still yield misleading conclusions. The final project looks at observational data with unobserved confounding and gives methods for computing bounds on average causal effects. Here, we discover some never-before-seen robustness properties unique to the partially-identified setting.




Estimating Causal Effects


Book Description

Explains the value of quasi-experimental techniques that can be used to approximate randomized experiments. The goal is to describe the logic of causal inference for researchers and policymakers who are not necessarily trained in experimental and quasi-experimental designs and statistical techniques.




Observation and Experiment


Book Description

Cover -- Contents -- Preface -- Reading Options -- List of Examples -- Part I. Randomized Experiments -- 1. A Randomized Trial -- 2. Structure -- 3. Causal Inference in Randomized Experiments -- 4. Irrationality and Polio -- Part II. Observational Studies -- 5. Between Observational Studies and Experiments -- 6. Natural Experiments -- 7. Elaborate Theories -- 8. Quasi-experimental Devices -- 9. Sensitivity to Bias -- 10. Design Sensitivity -- 11. Matching Techniques -- 12. Biases from General Dispositions -- 13. Instruments -- 14. Conclusion -- Appendix: Bibliographic Remarks -- Notes -- Glossary: Notation and Technical Terms -- Suggestions for Further Reading -- Acknowledgments -- Index




Causal Inference in Statistics, Social, and Biomedical Sciences


Book Description

This text presents statistical methods for studying causal effects and discusses how readers can assess such effects in simple randomized experiments.