U.S. General Accounting Office

Early Childhood Programs - The Use of Impact Evaluations to Assess Program Effects

Many researchers consider impact evaluations to be the best method for determining the extent to which the program itself, rather than other factors, is causing participant outcomes. Impact evaluations can be designed in several ways, but fall into two basic design categories: experimental and quasi experimental. Experimental designs randomly assign eligible individuals either to a group that will receive services from the program being studied or to a group that will not receive services from the program. The relevant outcomes of these two groups are measured and compared, and any differences found between the two can be attributed to the programs. While rigorous, these evaluations can raise ethical issues because they deny services to eligible individuals. This issue, along with other issues characteristic of complex evaluations, can make experimental impact evaluations especially difficult, time-consuming, and expensive to perform. Quasi experimental designs use a comparison group that is not created with random assignment. As a result, the comparison groups similarities to program participants are less certain. Quasi experimental designs are less rigorous, but tend to be less expensive and time-consuming and do not have the same ethical issues. 

The two federal programs focused most on early childhood education, Head Start and Even Start, are currently being studied using impact evaluations with experimental designs. Both of these programs are intended to produce improvements in children's school readiness and educational outcomes, including enhanced literacy. HHS is conducting two Head Start studies, both of which were mandated by the Congress. The first is a 6-year national impact study of 3-to-4-year-olds that will follow children through first grade. This evaluation will cost about $28.3 million. The second is of the Early Head Start program. This $21 million study will collect information about 17 local programs and follow children up to their third birthday. HHS has initiated another study that will follow these children through the spring before their entry into kindergarten. The $3.6-million Even Start study will examine the effectiveness of 18 local programs and will take 6 years to complete. The Congress mandated a study of Even Starts effectiveness but did not require that effectiveness be assessed using an impact evaluation. For the other nine programs we reviewed, impact evaluations are not currently being used to study the effectiveness of the early childhood education and care services they support. Researchers have used a variety of study designs, other than impact evaluations, to understand more about these types of programs with respect to early childhood education and care. 

HHS and Education promote and sponsor many types of research and evaluation studies. The value of a varied study agenda is that it provides agencies with answers to a broad range of questions about program operation and allows them to align research with the focus of the program. For example, HHS and Education sponsor evaluations that provide information on whether programs are being implemented as intended. The Early Head Start study currently under way includes an implementation study that provides information on the programs development and services provided to children and their families. For its CCDF program, which is aimed at increasing the availability, affordability, and quality of child care services HHS believes that increasing the level of knowledge and information about how child care systems work is a critical step toward improving the availability and quality of child care services. Therefore, HHS sponsors basic research to better understand the complex nature of child care markets at the state, local, and national levels.