K0SP 2045 - Causal Inference and Experimental Methods

The word “experiment” means different things in different contexts. If we were talking about art, architecture, or fashion, it would mean doing something a little unorthodox. But, if we are talking about science, as we will in this course, an “experiment” implies a research design and (usually) a set of statistical methods for testing causal claims. Across the sciences, experiments are the primary way to study cause and effect. Experimental methods take many forms in both the natural and social domains of scientific inquiry, but what unites them all is that a researcher carefully controls the presence (and absence) of some causal agent to measure its effect. In all of the sciences, the experimental method is the gold standard for causal inference. Randomized experiments take place in many settings: controlled laboratories, embedded within surveys, and embedded unobtrusively in people's natural environment. Each of these approaches comes with benefits and tradeoffs that affect the internal and external validity of the experimental design. This course is designed to introduce design and analysis principles for best research practices across the domain of experimentation. Experimentalists have been at the forefront of the “open science” movement in the social sciences that seeks to bring more transparency into the way we conduct and report research. We will devote some time discussing the origins of this movement as well as its aims and various proposals. Finally, because experiments generally rely on exposing human participants to different “treatments” it raises a set of ethical issues that researchers must consider before undertaking experimental research. We will discuss these as well.

Kevin ARCENEAUX

Séminaire

English

Autumn 2024-2025

Course Topics Session 1: The Fundamental Problem of Causal Inference This class will introduce students to the Neyman-Rubin Potential Outcomes framework for studying cause and effect, discuss common threats to causal inference in observational designs (e.g., collider bias, selection bias, endogeneity bias, etc.), and introduce the logic of randomized experiments in the social sciences. Session 2: Frequentist and Bayesian Approaches to Hypothesis Testing This class will review requentist hypothesis testing (e.g., p-values) and introduce them to the logic of Bayesian hypothesis testing. Session 3: The Open Science Movement This class will discuss questionable research practices that led to the “replication crisis” (“p-hacking,” “HARKing,” etc.) and various proposals from the proponents of the open science movement (preregistration, registered reports, etc.) Session 4: Ethics and Experiments This class will discuss ethical considerations when designing experiments (e.g., informed consent, deception, etc.) and the frameworks used for approaching these issues (e.g., the Belmont Report, Institutional Review Boards). Session 5: Primer on Experimental Design This class will introduce students to various experimental designs (e.g., between subject, within subject, factorial, etc.) and discuss the strengths and weakness of these designs. Session 6: Statistical Power and Effect Size This class will discuss statistical power (i.e., the sample size needed to detect a particular effect size). We will also consider various approach to thinking about and quantifying effect sizes (i.e., how “big” is the treatment effect in a substantive sense). Session 7: Lab and Survey-based Experiments This class will introduce students to the experiments conducted in labs and on surveys, focusing on the special issues involved when designing and analyzing these experiments (e.g., controlling the setting, measuring and maintain attention, ethical considerations). Session 8: Field Experiments This class will introduce students to experiments conducted in the field, focusing on the special issues involved when designing and analyzing these experiments (e.g., attrition, failure to treat, scalability, ethical considerations). Session 9: Natural and Qausi-Experiments This class will introduce students to natural and quasi-experiments, focusing on the special issues involved when designing and analyzing these experiments (e.g., as-if randomness, placebo tests, ethical considerations). Session 10: Testing for Heterogenous Treatment Effects This class will introduce students to moderators (manipulated and measured) and the approaches used to analyze their influence on treatment effects (e.g., interaction models). Session 11: Testing for Causal Mechanisms This class will introduce students to the mechanistic approach to thinking about causal effects (in contrast to the potential outcomes approach) as well as the strengths and weakness of statistical methods designed to study mechanisms (e.g., moderation models). Session 12: Interpretation of Causal Effects and Producing Reproducible Science The class will end on a discussion of how the experimental method does not free researchers from thinking deeply about how one should interpret the treatment effects uncovered in an experiment as well as ways to make our work reproducible.