K0SP 2045 - Causal Inference and Experimental Methods

This course equips students with the tools to design, implement, and analyze advanced experimental and quasi-experimental research. Weeks 1–3 cover core causal inference methods, including matching, regression discontinuity designs, instrumental variables, difference-in-differences, and synthetic control methods. Weeks 4–8 cover applied and advanced topics in experimental design, including survey and field experiments, pre-analysis plans, repeated-measurement designs, power analysis, and conjoint experiments. Weeks 9–12 focus on advanced data-science techniques and the use of large language models in experimental methodology. We will cover useful machine-learning methods (regularization and causal forests) and then explore the application of large language models to data cleaning, simulating human behavior, and building computational social-science tools for experimental research (e.g., chat applications). Throughout the semester, hands-on assignments and weekly in-class labs reinforce the material, culminating in a final essay in which you propose a study design.

To be specified by the lecturer.

PART I: QUASI-EXPERIMENTAL DESIGN (Weeks 1–3)

Focus: In this part of the course, we cover foundational causal inference methods for observational data. Emphasis is on understanding the assumptions behind each design, when they are credible, and how to implement them.

Week 1 – Re-cap of Causal Inference, Endogeneity, and Matching

• Topics: In this introductory class, we will briefly re-cap the logic of potential outcomes, counterfactuals, and the reasons behind randomization. We will also discuss design-based versus model-based inference. We then cover matching methods as a strategy for causal inference in observational data, including propensity score matching, coarsened exact matching, and their assumptions. Cunningham provides a good overview of regression and causal inference (Quant I and Quant II). Angrist and Pischke provide a similar overview more focused on experiments.
• Activities: Lecture & form groups of 3–4 people based on research interests.
• Readings:
• Scott Cunningham. “Causal Inference: The Mixtape”. Introduction and Chapter 2.
• Angrist and Pischke. Mostly Harmless Econometrics. Chapter 2.
• Cunningham. Causal Inference: The Mixtape. Chapter on Matching and Subclassification.

Week 2 – Regression Discontinuity Designs and Instrumental Variables

• Topics: This week covers two of the most important identification strategies in causal inference. We begin with regression discontinuity designs (RDD), both sharp and fuzzy, discussing the key assumptions (continuity of potential outcomes at the cutoff), bandwidth selection, and practical implementation. We then turn to instrumental variables (IV), covering the assumptions of relevance and exclusion, the Wald estimator, two-stage least squares, and the interpretation of the local average treatment effect (LATE).
• Activities: Lecture
• Readings:
• TBD

Week 3 – Difference-in-Differences and Synthetic Control Methods

• Topics: This week covers two closely related methods for causal inference in panel and time-series settings. We begin with difference-in-differences (DiD), discussing the parallel trends assumption, two-way fixed effects, and recent advances addressing staggered treatment adoption. We then turn to synthetic control methods, which construct a data-driven counterfactual from a weighted combination of untreated units. We discuss the method’s assumptions, inference via placebo tests, and extensions to multiple treated units.
• Activities: Lecture & in-class lab.
• Readings:
• TBD

PART II: ADVANCED EXPERIMENTAL DESIGNS (Weeks 4–8)

Focus: In this part of the course, we develop and apply advanced tools for survey and field experimental designs. Emphasis is on doing: you will draft pre-analysis plans, conduct power analyses, advanced randomization (including block/cluster and repeated-measures designs), and build survey tools.

Week 4 – Designing and Implementing Randomized Survey Experiments

• Topics: We discuss two common types of survey experiments that involve a treatment and a control group. The first are updating experiments, in which participants are asked about their prior beliefs before being presented with new information that may lead them to revise those beliefs. The second are information-provision experiments, where treated participants simply receive some form of factual or persuasive information. In lecture, we also examine the broader role of survey experiments in social science—whether to test psychological mechanisms, establish causal effects, or evaluate the effectiveness of specific treatments. Voelkel et al. illustrate the use of simple information-provision treatments within a megastudy framework, comparing the effectiveness of multiple interventions in parallel. Kamphorst and Willer provide an example of an updating experiment focused on correcting misperceptions.
• Activities: Lecture & in-class lab in groups. In the lab, we will first briefly go through some analysis code for survey experiments, and then we will use the rest of the time to work on the homework assignment. For the homework assignment, work in your groups of three. Everyone in the group will contribute one research question. For each idea, design some survey questions with your group for a specific sample of respondents. We will discuss each other’s research questions.
• Readings:
• EGAP guide on survey experiments.
• Voelkel et al (2024). Megastudy testing 25 treatments to reduce antidemocratic attitudes and partisan animosity. Science.
• Kamphorst and Willer (2024). Correcting misperceptions of the material benefits associated with union membership increases Americans’ interest in joining unions. PNAS.
• Homework assignment 1: You will program a survey in Qualtrics. It needs to contain one experiment. This can be an updating experiment (ask about peoples’ priors and update them) or a text-based experiment (provide people some information). Make sure your survey contains at least two open-ended questions! Your homework assignment will be completed with your group from week 1.

Week 5 – Statistical Power and Repeated Measurement Designs

• Homework assignment 1 is due.
• Topics: We discuss the concept of statistical power and how to use power analyses in your research. They’re particularly useful in two cases. First, you want to defend a null-effect and highlight that you had the power to detect something. Second, you want to estimate what sample size you will need. We then talk about experimental designs that increase your statistical power. EGAP introduces what power is. Clifford discusses several repeated-measurement designs. Broska et al provide an example of a repeated measurement design.
• Activities: Lecture & in-class lab. In the lab, you will perform a power-calculation on provided data, varying factors such as sample size, and you will analyze data from a repeated-measurement design (Broska et al).
• Readings:
• Clifford et al (2021). Increasing Precision Without Altering Treatment Effects. American Political Science Review.
• EGAP what is statistical power guide.
• Broska, Kamphorst, and Willer. Americans prefer predistributive over redistributive economic policies. Working paper.
• Homework assignment 2: You will program a repeated measurement design in Qualtrics. Use the experiment you programmed up for the first homework assignment and add at least one other treatment in the Qualtrics. Make sure you randomize the order of the treatments. In addition, take your own survey 10 times to create some data points that you can then use to write analysis code for your study. Make sure your code provides the ATEs and creates at least one plot visualizing the results.

Week 6 – Pre-Analysis Plans, Questionnaire Design, and Sampling Strategies

• Homework assignment 2 is due.
• Topics: This lecture we will focus on Pre-Analysis-Plans (PAPs). We will also talk about question design: what scales to use, and what are the benefits of each scale type? To be precise, we will discuss 0–100 vs shorter scales, and we’ll talk about trade-off scales. We also touch upon attention checks. Finally, we’ll discuss sample representativeness and talk about free samples. Berinsky discusses attention checks. Stagnaro discusses different online sample providers. Broockman is an example of a free sample that is (not) representative of the goal population.
• Activities: Lecture & Homework assignment. In class, you will first work-through any feedback I gave on your survey. We will then work on collecting some real data.
• Readings:
• EGAPs pre-analysis plans guide.
• Berinsky et al (2019). Using screeners to measure respondent attention on self-administered surveys. Political Science Research and Methods.
• Examples of simple and extensive PAPs (to be shared by instructor)
• Stagnaro et al (2024). Representativeness versus Response Quality: Assessing Nine Opt-In Online Survey Samples. Working paper.
• Broockman and Skovron (2018). Bias in perceptions of public opinion among political elites. American Political Science Review.
• Homework assignment 3: We’ll use your survey from homework assignment 1 (thus the survey without the repeated measurement design). First, work on your survey based on feedback you got in response to the previous homework assignments. Second, add an attention check to your survey and consider whether you want to change any of the scales in your survey. Third, make a Bovitz account with your group. Load some money on your account and collect X real responses to your survey. Fourth, based on those real responses, simulate the rest of the data as if you had collected data from 1000 respondents (no need to simulate the responses to open-ended questions). Next, write code that analyzes your data. Make sure your code contains at least one visualization. Finally, make a simple version of a pre-analysis-plan for your study (no need to pre-register it).

Week 7 – Conjoint Experiments: Structure, Implementation, and Analysis

• Topics: This week, we focus on conjoint experiments—a specific form of repeated-measures design widely used in social science. Conjoints are distinctive in that they yield a descriptive estimand: the Average Marginal Component Effect (AMCE), which captures how individual attributes influence choices on average. Hainmueller et al. (2017) provide a foundational introduction to the method and should be read in detail. Hainmueller and Hopkins offer an applied example. Leeper et al. highlight a common issue in analysis: how the choice of baseline category in your regression model can affect the interpretation. They propose the Marginal Mean as a more robust alternative. Heiss offers a practical guide—read this one most closely.
• Activities: Lecture & in-class lab where you will be working with conjoint data. In the second half of lab, we will have space for any question you have regarding your homework assignment.
• Readings:
• Hainmueller, Hopkins, and Yamamoto (2017). Causal Inference in Conjoint Analysis. Political Analysis.
• Hainmueller and Hopkins (2014). The Hidden American Immigration Consensus. American Journal of Political Science.
• Leeper, Hobolt and Tilley (2019). Measuring Subgroup Preferences in Conjoint Experiments. Political Analysis.
• Bansak et al (2017). The Number of Choice Tasks and Survey Satisficing in Conjoint Experiments. Political Analysis.
• Andrew Heiss’ conjoint guide.

Week 8 – Conducting Field Experiments

• Homework assignment 3 is due.
• Topics: This week, we turn to field experiments. For PhD students, they offer the advantage of being implementable with minimal funding (in some cases), yet they often demand significant time and logistical effort, carry a high risk of failure, and can be difficult to publish. We will focus on the mechanics of field experiments, with particular attention to different randomization strategies, including block and cluster designs. We will also discuss where these studies most often fail: in the implementation stage.
• Activities: Lecture & Group activity. In your groups, you will develop a proposal for a field experiment. We will then discuss each group’s idea together, focusing on feasibility, design choices, and potential pitfalls.
• Readings:
• Gerber and Green. Field Experiments. Chapters 3 and 4.
• EGAP randomization guide.
• Mousa (2020). Building social cohesion between Christians and Muslims through soccer in post-ISIS Iraq. Science.

PART III: MACHINE LEARNING AND LLMS IN EXPERIMENTAL METHODOLOGY (Weeks 9–12)

Focus: In this part of the course, we explore how machine learning methods and large language models (LLMs) can be used to extend experimental research. You will learn how to apply regularization techniques and causal forests, and how to use LLM APIs for tasks like data cleaning, text classification, behavioral simulation, and building computational tools such as chat applications. As before, the emphasis is on doing: hands-on labs and assignments will guide you through implementing these methods on your own data.

Week 9 – Machine Learning Essentials: Regularization Methods (LASSO and Ridge Regression), Causal Forests and Exploring Heterogeneous Treatment Effects

• Topics: This week, we will cover machine learning methods commonly used in social science research, with a focus on techniques that are both interpretable and practically useful for experimental analysis. Regularization methods—such as LASSO and Ridge regression—are particularly useful when dealing with many covariates or multicollinearity. Causal forests are designed to detect and model heterogeneous treatment effects. They automate the search for subgroups that respond differently to a treatment, offering a data-driven approach to exploring effect heterogeneity.
• Activities: Lecture & short lab. In the lab you will run these models yourself.
• Readings:
• TBD
• TBD

Week 10 – Using Language Model APIs: Data Cleaning, Chat Applications, and Other Uses

• Topics: This week covers LLM APIs and their applications in social-scientific research. We begin with the basics of setting up the OpenAI API and using it for data cleaning and classification—the simplest and most common use-case. We then turn to more advanced applications: chat applications and content manipulation tools. Mernyk, Kamphorst, and Willer describe a case in which LLM-based chat functionality is used as a Voting Advice Application. Piccardi et al. explore how LLMs can modify digital environments, for example by altering social media feeds. Gilardi and Mellon focus on classification.
• Activities: Lecture & in-class lab. You’ll work with the ChatGPT API to classify the open-ended responses that you collected for the third homework assignment.
• Readings:
• Gilardi et al (2023). ChatGPT outperforms crowd workers for text-annotation tasks. PNAS.
• Mellon et al (2024). Do AIs know what the most important issue is? Research & Politics.
• Piccardi et al. Social Media Algorithms Can Shape Affective Polarization via Exposure to Antidemocratic Attitudes and Partisan Animosity. Working paper.
• Mernyk, Kamphorst, and Willer. Large Language Model powered voter guides can reduce informational barriers to voting. Working paper.
• Homework assignment 4: Use the data you collected for homework assignment 3. First, use the LLM API to classify the open-ended responses from the human data. Second, write a short two-page report summarizing your findings from the data you have collected so far. Your report should contain: the main effect of your experiment, at least one visualization, and a discussion of the LLM classification results.

Week 11 – Simulating People with LLMs

• Homework assignment 4 is due.
• Topics: This week we talk about using Large Language Models to simulate humans. DellaVigna and Broockman provide context about why this may be useful: experts (you and I!) are bad at predicting human behavior. The other readings discuss different ways to simulate human behavior and attitudes.
• Activities: Lecture & Group discussion. In the discussion we will talk about potential pitfalls and opportunities of these methods.
• Readings:
• DellaVigna and Vivalt. Forecasting Social Science: Evidence from 100 Projects. Working paper.
• Broockman et al. Political practitioners poorly predict which messages persuade the public. PNAS.
• Park et al. Generative Agent Simulations of 1,000 People. Working paper.
• Ashokkumar et al. Prediction of Social Science Experimental Results Using Large Language Models. Working paper.
• Kamphorst et al. Generative Agents Accurately Predict Behavioral Effects in Field Experiments. Working paper.
• Homework assignment 5: Design different versions of the treatment you reported on in homework assignment 4. Test these different versions of the treatment using an LLM method.

Week 12 – Group Presentations

• Homework assignment 5 is due.
• Activities: Each group will prepare a short presentation discussing the findings of their experiment. After each presentation, we will briefly brainstorm with the class how we could have made each experiment even stronger.
• Readings:
• The reports from all the groups.

Homework assignments (60%) – Continuous assignments throughout the semester.

Group presentations (15%) – Group presentations.

Final assignment (25%) – A design plan for a study that incorporates the topics we have discussed from at least three of the weeks.

The syllabus provides a brief overview of each assignment. For detailed instructions, please consult the dedicated assignment instructions, which will be distributed by the instructor during class. Most homework assignments will be completed in small groups based on shared research interests. Over the course of these assignments, you will design your own survey, collect your own data, analyze it, and simulate respondents using large language models (LLMs). There are five homework assignments in total. While we’ll work on many of them together during class, they are still time-intensive.

You may use either R or Python for your assignments. Submissions can be in the form of R Markdown files (or similar formats that show code) or Jupyter notebooks. Qualtrics files should be shared with me by inviting me to the Qualtrics survey as a collaborator.

You are welcome to use LLMs to assist with your homework assignments. However, you should be able to understand the code they generate. I may occasionally ask you to explain parts of the code you submit. This is intended to encourage you to either ask ChatGPT for explanations or to prompt it to write simpler, more understandable code. Further, as per Sciences Po’s policies, if you use an LLM, you need to indicate that you did so.

Finally, be aware that these assignments are challenging. Expect to encounter obstacles and for some things not to work out as planned. You’ll be graded not on the success of your final product, but on the ambition and thoughtfulness of your approach.