Modern policies are commonly evaluated not with randomized experiments but with repeated measures designs like difference-in-differences (DID) and the comparative interrupted time series (CITS). The key benefit of these designs is that they control for unobserved confounders that are fixed over time. However, DID and CITS designs only result in unbiased impact estimates when the model assumptions are consistent with the data at hand. In this paper, we empirically test whether the assumptions of repeated measures designs are met in field settings. Using a within-study comparison design, we compare experimental estimates of the impact of patient-directed care on medical expenditures to non-experimental DID and CITS estimates for the same target population and outcome. Our data come from a multi-site experiment that includes participants receiving Medicaid in Arkansas, Florida, and New Jersey. We present summary measures of repeated measures bias across three states, four comparison groups, two model specifications, and two outcomes. We find that, on average, bias resulting from repeated measures designs are very close to zero (less than 0.01 standard deviations; SDs). Further, we find that comparison groups which have pre-treatment trends that are visibly parallel to the treatment group result in less bias than those with visibly divergent trends. However, CITS models that control for baseline trends produced slightly more bias and were less precise than DID models that only control for baseline means. Overall, we offer optimistic evidence in favor of repeated measures designs when randomization is not feasible.

中文翻译：

---

重复测量设计的因果主张的有效性：差异中的差异和比较中断时间序列的研究内比较评估。

现代政策通常不是通过随机实验来评估，而是通过重复测量设计，例如双重差分（DID）和比较中断时间序列（CITS）。这些设计的主要优点是它们可以控制随着时间的推移而固定的未观察到的混杂因素。然而，当模型假设与手头的数据一致时，DID 和 CITS 设计只能得出无偏影响估计。在本文中，我们通过实证检验在现场设置中是否满足重复测量设计的假设。使用研究内比较设计，我们将针对相同目标人群和结果的以患者为导向的护理对医疗支出影响的实验估计与非实验 DID 和 CITS 估计进行比较。我们的数据来自一项多地点实验，其中包括在阿肯色州、佛罗里达州和新泽西州接受医疗补助的参与者。我们提出了三个州、四个比较组、两个模型规范和两个结果的重复测量偏差的汇总测量。我们发现，平均而言，重复测量设计产生的偏差非常接近于零（小于 0.01 标准差；SD）。此外，我们发现，治疗前趋势与治疗组明显平行的比较组比具有明显不同趋势的比较组产生的偏差更小。然而，与仅控制基线均值的 DID 模型相比，控制基线趋势的 CITS 模型产生的偏差稍多，且精度较低。总体而言，当随机化不可行时，我们提供了支持重复测量设计的乐观证据。