Ground Truth.
AI, checked against the source.

Difference-in-Differences: Estimating Effects When Adoption Is Staggered

2026-11-18 · Breach Protocol: Inside the AI Blackbox — full transcript

Learn how difference-in-differences uses changes in a comparison group to estimate an intervention's effect, why adoption at different times complicates the calculation, and how to assess the assumptions behind an estimate. This audio was created with Boson AI's Higgs Audio — https://www.boson.ai/higgs-audio

Listen on Spotify · Watch on YouTube · Pocket Casts

What a before-and-after change can tell us

Eris: If firms complete more work after a training program, the increase alone doesn't tell us how much the training caused. Difference-in-differences estimates the program's effect by comparing those firms' change with a second group's change over the same period.

Vestra: <|prosody:expressive_high|>Take a hypothetical set of firms that introduces a training program. The intervention is the thing whose effect we want to know: the program. The outcome is what we measure, perhaps the average number of completed tasks per worker each week. Suppose that average goes from ten to fifteen after training. <|prosody:pause|>That's a rise of five, but demand, staffing, or other conditions may also have changed.

Eris: That's what bothers me about giving the training all the credit. The causal question is narrower: how much of that rise came from the training? We'd need to compare the trained firms with what those very same firms would have done in the same period without training. That missing alternative is called the counterfactual. We cannot observe both versions of the same firms at the same time. How do we estimate the change we can't see?

Vestra: Difference-in-differences uses another set of firms to estimate the missing change. The treated group receives the intervention; the comparison group, sometimes called the control group, does not receive it in the period being compared. Suppose the comparison firms go from twenty tasks per worker to twenty-two. Their change is two. The trained firms' change is five. <|prosody:pause|>Five minus two gives an estimated effect of three tasks per worker.

Eris: The two-task rise is the comparison group's observed change. If the trained firms would also have risen by two without training, their counterfactual would be twelve, and the observed fifteen is three above it. <|prosody:expressive_high|>That last sentence contains the assumption the arithmetic needs. The comparison group doesn't automatically reveal the missing future.

The parallel-trends assumption

Vestra: <|prosody:expressive_high|>The key assumption is called parallel trends. It says that without the intervention, the treated group's average outcome would have changed by the same amount as the comparison group's average over the relevant period. <|prosody:pause|>It concerns the change, not the starting level. In our hypothetical, the firms can start at ten and twenty tasks: under the assumption, both would rise by two without training.

Eris: <|emotion:relief|>That makes the different starting points less puzzling. So the lines for the two groups need not lie on top of each other. Their gap would have stayed the same in the no-training world. The method allows stable differences between the firms, but a new event affecting one group more than the other can still be mistaken for training's effect.

Vestra: And that new event is the part I'd worry about. For example, suppose trained firms also receive a new source of demand at the same time, and that demand raises their task counts independently of training. The comparison firms' two-task increase would not capture the whole change the trained firms would have experienced without training. The difference-in-differences estimate would include some effect of that extra demand.

Eris: I'd want to see how those firms were changing before training, too. Researchers often inspect the groups before the intervention. If their changes already diverged, that can raise a concrete concern about the comparison. <|prosody:pause|>But even a flat-looking difference before training cannot prove the missing post-training trend would have been parallel. The groups could diverge later, and a small or noisy dataset can hide earlier differences. And what counts as before training?

Vestra: <|sfx:humming|>Hmm, another assumption concerns anticipation: firms may change behavior because they know training is coming. If they reorganize work before the stated start date, the supposedly untreated baseline may already reflect the program. Researchers can choose an earlier baseline if they have a defensible account of when anticipation begins, but that choice comes with assumptions too.

When adoption happens at different times

Eris: <|prosody:expressive_high|>Now let the hypothetical training program arrive in different years. Some firms start in year two, some in year three, and some never adopt during the study. That's staggered adoption: the intervention starts at different times for different groups. Callaway and Sant'Anna study this setting when treatment, once started, remains on.

Vestra: To estimate an effect for the year-two adopters in year two, a researcher can compare their change since year one with the change for firms that have not yet adopted. Those could include the year-three adopters and the never-adopters. I like being able to name who supplies that comparison. The comparison requires a defensible parallel-trends assumption for those groups, and the supposedly untreated firms must not already be responding to the coming program. What happens to that comparison when the year-three firms adopt?

Eris: Once the year-three firms adopt, they are no longer untreated comparisons for later periods. A never-treated group may still be available, but it has to be a credible comparison too. <|prosody:pause|>If everyone eventually adopts, the pool of untreated comparisons disappears. The last adopters and later effects cannot be estimated using this particular untreated-comparison strategy once there is no suitable untreated group left.

Vestra: There may also be differences in observable characteristics, such as firm size, that are related to how outcomes change. Researchers can compare firms with similar pre-intervention characteristics, or adjust for those differences using a model.

Eris: <|sfx:humming|>Hmm, how much reassurance should we take from that adjustment?

Vestra: That replaces the original assumption with a conditional version: among comparable firms, their untreated changes would be parallel. <|prosody:pause|>It does not remove unobserved changes that affect the groups differently, and the data need enough comparable firms in both groups.

Why a familiar regression can mix comparisons

Eris: A regression is a statistical way to describe how an outcome varies with several recorded features. A common version for this problem is called a two-way fixed-effects regression. It accounts for each group's stable average differences and for changes common to each time period, then summarizes the association with treatment.

Vestra: With a single adoption date and a valid comparison, that regression can reproduce the basic difference-in-differences calculation. <|prosody:expressive_high|>With staggered adoption, the conventional regression can also use an early adopter as a comparison for a later adopter, even though the early group is already treated. If the early group's treatment effect changes between those periods, its outcome change contains that changing effect. Subtracting it can distort the later group's estimate.

Eris: That already-treated comparison worries me. For a concrete hypothetical, imagine the later group gains five tasks per worker after training, while the early group's outcome gains two more because its own training effect is still growing. Comparing those changes yields three, even if the later group's true effect is five and their untreated trends would otherwise have matched. The supposedly neutral comparison has moved because of its own intervention.

Vestra: <|prosody:expressive_high|>When effects differ across groups or over time, a conventional combined estimate can give some underlying effects negative weight, and in extreme cases <|prosody:pause|>even point in the opposite direction from all of them.

Eris: <|emotion:surprise|>The opposite direction is a nasty surprise. What happens when the regression tracks the periods before and after adoption?

Vestra: Sun and Abraham also show that conventional regressions tracking time before and after adoption can mix effects from other periods into a given estimate. A supposed pre-treatment signal can arise from later treatment effects, even under their parallel-trends and no-anticipation assumptions.

Eris: I wouldn't want the fix for that regression problem to give us false reassurance. That is a separate problem from whether the groups would have followed parallel trends. A clearer estimator can avoid those mixed comparisons while still depending on parallel trends and the other assumptions. Neither issue makes the other disappear.

Estimating effects by group and time

Vestra: <|emotion:enthusiasm|>Callaway and Sant'Anna's approach starts with a more specific question: for firms that first adopted in a particular year, what was the average effect in a particular later year? That's a group-by-time average treatment effect. For each such estimate, the method compares that adopting group's change from a suitable pre-treatment period with the change for an eligible untreated comparison group.

Eris: I like having a specific group and year to attach the answer to. Their R software tutorial demonstrates the workflow with simulated data, where the authors set the answer in advance. It records each unit, time, outcome and first treatment date; estimates a set of group-by-time effects against never-treated units; and then combines those estimates. The simulated result follows the increasing effect built into that example. <|prosody:pause|>That's a demonstration under known conditions, not proof that those conditions hold in a real study.

Vestra: <|prosody:expressive_high|>One useful summary is an event study: it aligns groups by time since adoption, rather than by calendar year. It can average the estimates in the first treated period, then those one period after adoption, and so on. That makes timing visible, but the average at each point may include different groups. Late adopters have less follow-up, so an apparent change across the graph can partly reflect who is included.

Eris: <|emotion:surprise|>That would bother me when reading the curve. Can we keep the groups the same?

Vestra: The tutorial offers a way to hold the included groups fixed over a chosen follow-up window. That helps distinguish a change in effects over time from a change in group composition, but it excludes groups without enough follow-up. <|prosody:pause|>The resulting answer describes that smaller set. Different choices of weights and averaging can also answer different questions, even with the same underlying estimates.

Eris: So I'd want to know which groups that choice leaves out. And how do we show how uncertain the estimates are?

Vestra: Estimates have uncertainty from the data. A confidence interval gives a range calculated to reflect sampling uncertainty under the method's assumptions; <|prosody:pause|>it does not measure every possible failure of the comparison. The tutorial also shows bands designed for examining a whole collection of estimates together. Small groups, weak overlap between treated and comparison firms, or too few untreated groups can make estimates imprecise or unavailable.

Reading the causal claim

Eris: <|prosody:expressive_high|>When reading a staggered difference-in-differences study, first identify who adopted what, and when. Then ask which untreated or not-yet-treated groups supply the comparison at each time, and whether their circumstances make the required parallel trends plausible. An adoption date alone doesn't make a group a good comparison.

Vestra: And I'd resist treating a quiet-looking pre-intervention graph as reassurance on its own. Look for anticipation and other changes occurring around adoption. Check the pre-intervention evidence, including its uncertainty: a visible divergence can challenge the design, while no detected divergence does not prove the missing post-intervention trend. With staggered adoption, check how pre-period estimates were constructed. Conventional event-study regressions can mix in post-treatment effects when effects vary across groups or over time. Some specifications can also be affected by effects in the periods left out as reference points.

Eris: <|prosody:expressive_high|>I'm also going to be more careful with those long-run points. Finally, ask what outcome is measured, which population the estimate covers, how the group-and-time estimates are weighted, and whether the long-run points represent only early adopters. Those details define what the reported average actually describes.

Vestra: <|prosody:expressive_high|>Staggered difference-in-differences is an observational design: researchers use data on adoption that occurred without necessarily assigning it at random. It can give an informative estimate of an intervention's effect when the comparison assumptions are defensible. <|prosody:pause|>The calculation alone cannot guarantee that the observed difference was caused by the intervention.

Sources and follow

Vestra: Which comparison would you want explained before trusting a training claim? Leave us a comment.

Eris: <|emotion:contentment|>You can find more reporting at Ground Truth, groundtruth.day, and hear the day's stories in the separate news brief. Follow AI in Detail for more explanations.