The Complex Challenges of Measuring the Impact of Social Programs

As efforts to evaluate housing programs illustrate, it's difficult to make precise comparisons. But it's worth the effort.
April 20, 2016
Feather O'Connor Houstoun
By Feather O'Connor Houstoun  |  Contributor
A senior adviser to the Wyncote Foundation

When is a comparison group not a comparison group? In social-science research, figuring that out may mean the difference between conclusions worth building programs on. A recent economic analysis in the New York Times examining anew a decades-long initiative helping low-income families move from high-poverty areas demonstrates how challenging program-impact evaluations can be. But greater understanding of how to compare may lead to more informed policy.

Dorothy Gatreaux was an African-American mother living in Chicago's highly segregated public housing who successfully sued the Chicago Housing Authority for racial discrimination. The resulting 1976 consent decree created a voucher program that gave 7,500 low-income families the opportunity to move to less racially segregated, more affluent neighborhoods in the city and its suburbs. (I worked in the U.S. Department of Housing and Urban Development's headquarters and provided program support in the settlement negotiations.)

The improved outcomes for families using the voucher program, especially for those moving to suburban areas, was stunning, and Moving To Opportunity, a Gatreaux-inspired national program, was created by HUD in the 1990s to operate in five metropolitan areas.

So how have things worked out? Following family outcomes over time is always challenging and susceptible to misleading conclusions. Gatreaux did not have a non-participating control group, but did track outcomes for subgroups located in suburban and urban neighborhoods. The research that followed participants in Moving to Opportunity, and a second wave called Gatreaux 2, established control groups not participating in the program and collected more detailed family data. But researchers examining all three programs were confounded by program design changes. Moreover, each program was launched in a different economic climate, and the advent of welfare reform in the late 1990s influenced workforce behavior significantly for movers as well as for those staying in traditional public housing.

That's not to say that nothing has been learned. The longitudinal story of how these families fared -- in school achievement, mental well-being, employment, and other social indicators -- has been well-researched, with mixed conclusions about the value to families of moving out of high-poverty public housing into more diverse neighborhoods. The most recent study found that young children who moved out of public housing had far higher earnings than similar children whose parents were not chosen in the lotteries to receive vouchers. Outcomes for both sets of parents were essentially the same, suggesting that the benefits of moving into a less-poverty-impacted neighborhood had the greatest effect during formative developmental years.

But Erin Chyn, a doctoral student at the University of Michigan, discovered that there was more to be found in the data. Moving to Work families were chosen via a lottery, which created winners and losers, the latter serving as a control group. However, only a fourth of eligible families chose to apply for the lottery. Chyn hypothesized that families that were highly motivated to escape poor neighborhoods and to provide better environments for their children were more likely to apply, thereby flattening somewhat the differences between the lottery winners and losers.

Chyn turned to another program operated by the Chicago Housing Authority to identify a more randomized comparison of neighborhood impacts. In the late 1990s, CHA began a large-scale demolition of family high-rise developments. Residents of buildings selected for demolition received vouchers; those whose buildings were undisturbed did not. This process produced large numbers of families in equal circumstance being treated differently. Chyn found that the children forced to move went on to have annual earnings 16 percent higher and a 9 percent greater employment rate. Yet a contemporaneous CHA program offering vouchers to volunteer families had no similar result.

Voucher programs are now well-established mainstream housing-assistance programs, but have evolved far from the original directive that families be helped to move into more diverse neighborhoods. Given the prevailing controversies competing for attention, it's unclear whether more recent research may propel a retooling of housing-voucher practice.

Nonetheless, more-refined family-outcomes research should inform assessment of other public programs that offer choices to families and purport to produce better outcomes. Traditional charter-school enrollment, for example, begins with the active search by a parent for a school of choice for the family's children. What influence does those parents' engagement have on the results of the school? How do they differ in result from conversions of entire publicly-run school systems to charter operators, where families have not chosen to enroll their children in charters?

Thanks to decades of research, we are in a better position to answer such questions with confidence.