Child Care and Early Education Research Connections

Skip to main content

Generalizing treatment effect estimates from sample to population: A case study in the difficulties of finding sufficient data

Description:
Background: Given increasing concerns about the relevance of research to policy and practice, there is growing interest in assessing and enhancing the external validity of randomized trials: determining how useful a given randomized trial is for informing a policy question for a specific target population. Objectives: This article highlights recent advances in assessing and enhancing external validity, with a focus on the data needed to make ex post statistical adjustments to enhance the applicability of experimental findings to populations potentially different from their study sample. Research design: We use a case study to illustrate how to generalize treatment effect estimates from a randomized trial sample to a target population, in particular comparing the sample of children in a randomized trial of a supplemental program for Head Start centers (the Research-Based, Developmentally Informed study) to the national population of children eligible for Head Start, as represented in the Head Start Impact Study. Results: For this case study, common data elements between the trial sample and population were limited, making reliable generalization from the trial sample to the population challenging. Conclusions: To answer important questions about external validity, more publicly available data are needed. In addition, future studies should make an effort to collect measures similar to those in other data sets. Measure comparability between population data sets and randomized trials that use samples of convenience will greatly enhance the range of research and policy relevant questions that can be answered. (author abstract)
Resource Type:
Reports & Papers
Country:
United States
State(s)/Territories/Tribal Nation(s):
Pennsylvania

- You May Also Like

These resources share similarities with the current selection.

Module 4: Sampling weights and variance estimation

Other

Decomposing treatment effect variation

Reports & Papers

Sampling weights and variance estimation [Powerpoint]

Other
Release: 'v1.57.0' | Built: 2024-03-14 09:29:08 EDT