KSOD 2600 - Statistical text analysis

Objectives. Text is an important form of evidence in the social sciences. Novel methods for studying features of text quantitatively have developed rapidly in recent years, and applications of these tools to previously intractable research problems are among the most exciting growth areas in the social sciences. This course is designed for M2 students who have taken introductory coursework in quantitative methods and wish to extend their skillset to incorporate modern statistical text analysis methods. The goal of the course is to help students develop as informed and critical users of these methods. By the end of the course, you will have a strong understanding of the key epistemological challenges raised by studying language use, as well as a deeper grasp of how statistical text analysis methods extend familiar statistical tools (i.e., linear regression) to address these challenges. Assignments will emphasize building a basic text analysis pipeline that can be used in future research projects.

Prerequisites. Students should have completed two semesters of coursework in quantitative methods (probability theory, correlation, ordinary least squares, generalized linear models) involving some practical data analysis experience (using e.g. R or Python).

Evaluation. Students will complete a research project that takes a corpus of text and uses a modern language model to describe its features. The project is primarily intended to help students whose mémoires involve a text analysis component, but other projects are welcome; students are encouraged to consider projects at a scale appropriate to the length of the term.

Course organization. The course is split into three parts.

● In Part 1, we will discuss the basic opportunities and challenges of using text as data in the social sciences. Our focus in this section will be primarily conceptual; we will focusing on asking and clarifying questions about text corpora, and we will spend some time discussing the theoretical basis for approaching semantic analysis as a quantitative problem. We will also spend some time at the beginning of the term setting up a text analysis pipeline on your computer that you can work with throughout the semester.

● In Part 2, we will develop the mathematical infrastructure needed to understand how machine learning methods infer useful patterns from data, with a strong emphasis on the application of these ideas to text. We will focus in particular on the statistical concepts of information and predictability and their quantitative representations.

● In Part 3, we will focus on the design and use of large language models. We will compare the benefits and drawbacks of a few ways of using LLMs (e.g. supervised fine tuning, prompt-based methods), and we will also consider some recently proposed frameworks for using LLMs in the social sciences.

Schedule of topics:

Part 1: Asking questions about text

• Course overview; choosing a corpus; technical setup.
• What can we learn from text? Word frequencies and the distributional hypothesis

Part 2: Technical foundations

• Probability theory and information theory
• Matrix analysis and linear models
• Machine learning
• Principal component analysis

Part 3: Modern text analysis

• Word embeddings
• Large language models (part 1)
• Large language models (part 2)