FacebookTwitterEmail

“Wherever possible, we should design new initiatives to build rigorous data about what works and then act on evidence that emerges—expanding the approaches that work best, fine-tuning the ones that get mixed results, and shutting down those that are failing.”
­—Peter Orszag, former director of the U.S. Office of Management and Budget[1]

Any standard course in policy analysis will typically include a lengthy discourse about the importance of data and evidence. When policymakers or analysts face a problem, data can play at least four key roles in their decision-making process:

  • Problem definition: Data can be used to focus attention on the precise problem policymakers are interested in solving;
  • Option-building: Data can be used to identify the set of policy interventions that can have an impact on the problem;
  • Prediction: Data can be used to predict how a particular policy intervention is likely to change conditions on the ground if implemented in a certain context;
  • Evaluation: Data can be analyzed to establish whether a particular policy intervention has helped improve the situation.

These actions can generate a cache of evidence to help inform policy decisions to increase or reduce the scale and scope of a policy or program, modify a program’s structure or incentives, eliminate a policy or program altogether, or introduce a new policy or program. This is classic policy analysis.

However, policymaking in the United States has not always followed this “textbook” approach, and consensus is emerging that policy would be more effective if evidence were more regularly brought to bear on key policy questions. This belief helped motivate the Obama Administration’s multipronged efforts to promote evidence-based policy at the federal level.[2] The essays in this book focus on creating preconditions so that: (1) the right data are available for policy analysts to conduct problem definition, option-building, prediction, and evaluation; and (2) the right lessons are gleaned from these analyses. These are the building blocks of evidence-based policymaking.

However, the record of evidence translating cleanly into policy is not as stellar as it should be. For example, most scientists agree that the evidence is clear regarding the human role in contributing to climate change. A joint National Academy of Sciences and Royal Society report in 2014 summarizes the evidence and shows a direct correlation between the rise in planetary temperatures and more intensive human use of fossil fuels.[3] And yet this evidence has made limited inroads, at best, where new policy is concerned. A second example is in transportation planning. It is widely recognized that light rail projections used by transportation officials routinely overestimate ridership and underestimate the cost of constructing light rail systems.[4] However, these projections are rarely adjusted and the erroneous projections still make the news, as if the variances were truly unexpected. In both examples, evidence has not translated into policy.

These cases reveal a simple truth: Developing evidence is not a sufficient condition for implementing evidence-based policy. More is needed. This discussion focuses first on the evidence that is useful for policymaking. Then it turns to the roles that “narrative” and “vehicle” can play in creating an environment in which evidence is recognized, understood, and incorporated into policy discussions and debates. A narrative is a simple, personal story that captures the relationship in a way the average person can understand. A vehicle is a conduit, such as a newspaper, through which the narrative is delivered. But even a strong narrative and vehicle cannot guarantee that evidence is incorporated into policy. Other elements, such as a focusing event that garners attention to a policy problem and an absence of gate-keepers committed to maintaining the status quo, are critical.[5] Unfortunately, researchers and others pay far less attention to creating a compelling narrative and vehicle than they do to ensuring that the latter factors are in place. The result is less use of evidence in policymaking.

What Counts as Evidence?

A precondition for injecting evidence into the policy-making process is the existence of evidence that can potentially inform policy. There has been an ongoing debate about what constitutes actionable evidence. There is general consensus that the clearest evidence emerges from randomized, controlled trials (RCTs) in which people are randomly assigned to either a treatment or control group, and the study environment is closely managed or tracked to ensure that all other factors that could affect outcomes are controlled for. However, broad execution of RCTs within the social sciences is impractical because they are often difficult to design and expensive to run. Furthermore, RCTs raise ethical questions. How do you not offer a high-quality classroom experience, for example, to children in a control group? As such, few RCTs are undertaken in the social sciences.[6]

This leaves us in a world woefully short of “gold-standard” evidence in many policy areas. As a result, policymakers and analysts who insist that the results of RCTs studies are the only valid form of evidence too often have nothing to use to support a position for changing policy, even when problems with particular policies are acknowledged. Strict adherence to an RCT−only view, therefore, can result in making decisions based on less information and evidence than is available, ultimately resulting in less evidence-based policy. This can also lead to a bias to preserve the status quo—no evidence equals no possibility of improvement.

But often other information can be brought to bear in policy areas lacking RCTs. There are many other types of high-quality studies that use valid data and sophisticated statistical methods that control for potentially confounding factors. I find these “imperfect” high-quality studies to be informative and useful as evidence, but I understand the reluctance of some in the research, policymaking, and research funding communities to embrace them. However, this hesitance need not result in policy stalemate, where one group says we have evidence and another says we do not. An underused statistical approach, known as meta-analysis, may be helpful in this regard. Meta-analysis synthesizes the findings of a set of research studies, which can be insightful, even when none of the studies is a randomized, controlled trial. It allows one to argue that “the preponderance of evidence suggests” using unbiased statistical techniques and thus can help build a policy consensus. While beyond the scope of this discussion, I would encourage more assessments of research on a given policy area using meta-analysis as a supplement to randomized, controlled studies, in addition to more support for those proposing such pursuits.

The use of meta-analysis would certainly have been helpful to the City of Fresno, CA. In 2012, city officials in Fresno embarked on an extended debate about whether to privatize their waste management services as part of a fiscal belt-tightening. The debate produced a divided city council, a 4−3 council vote to privatize the service, and then a citywide referendum reversing the privatization decision. The result: No change in service, hard feelings throughout the community, and tens of thousands of already-scarce dollars spent on the referendum vote rather than providing services to residents. The problem: The argument focused on the wrong issue, privatization. If the city had used results from an existing meta-analysis conducted by academic researchers, they could have clarified that the issue was not whether the service was provided by a public or private entity, but whether providers must compete for the franchise.[7] Competitive tendering leads to cost improvements regardless of whether the awardee is a private company or a public agency. This evidence would have made a big difference in Fresno, and left the community stronger.

But meta-analysis alone is not enough. We also need more policy experiments that generate information on observed effects from which we can glean insights into how programs and incentives work in practice. Two related examples from my former agency, the Department of Housing and Urban Development (HUD), highlight programs that represent such experiments at the federal and local levels. On the federal level, the Moving to Work (MTW) program allows local public housing authorities, with HUD approval, to modify some operating guidelines in rental assistance programs to lower costs and promote self-sufficiency among residents. There are now more than 35 MTW housing authorities, and they have instituted dozens of new program policies. Sadly, the follow-through on evaluating these changes has not been as robust as one would like. But there remains an opportunity to learn much.

The way the Denver, CO, Public Housing Authority (PHA) manages its portfolio exemplifies a policy experiment by local governments. The PHA maintains a public housing portfolio that has two distinct configurations. One portion of the portfolio consists of large block of units located at a single site—the quintessential image of public housing. A second portion consists of individual or small sets of units scattered throughout the PHA service area. Denver’s policy for new recipients was to randomly assign them to either the concentrated block of units or scattered site housing. This distribution offers a natural experiment that allows for the policymaker to assess the effects of concentrating rental assistance units, with potential implications for how best to maintain and adjust public housing portfolios to increase residents’ quality of life and improve outcomes for program participants. This experiment differs from the large-scale demonstrations, such as HUD’s 20-year Moving to Opportunity demonstration, that include a purposeful decision to implement a research design. Here the Denver PHA simply implemented their program to mimic a research design, which provides high-quality insights. I believe there are many more such natural experiments in the field.

Using Evidence Effectively: Narrative and Vehicle

Although evidence is the precondition in evidence-based policymaking, two other tools are required: a narrative and a vehicle. Too often, evidence is presented and made available in lengthy academic documents that appeal to only researchers and academics. Policymakers rarely have the training or the time to sift through such documents to fully digest the results. What they need is a narrative, a concise short story that presents the evidence in a way that is memorable and intuitive. The narrative serves as a shorthand distillation and translation of the compiled evidence and becomes the embodiment of the lessons learned and actions to be taken. The most effective narratives will include clear explanations of directly-supporting evidence. But the story leads with the narrative, not the data and evidence.

An appropriate vehicle for delivering the narrative is also essential for the effective implementation of evidence-based policy. We all have read a good book or short story and wondered why it didn’t gain traction. One possibility might be that the author or publisher didn’t promote the work in the most powerful way. The same challenge can arise for evidence and a narrative. It is not enough to publish significant results of studies in academic journals or publications. When the vehicle for the narrative is not on policymakers’ radar, it is hard to inject evidence into policy.

What represents the ideal combination of narrative and vehicle? It depends on the audience. A different approach is needed if the intended audience is composed of key lawmakers, leaders, and staff who have an ability to shape legislation and policy or if the target audience is the general public or the social circles of the key lawmakers, leaders, and staff.

To target key players, the vehicle should be a short document (one to two pages) with main conclusions from the evidence laid out clearly and concisely. The document may be slightly longer if it is generated “on the inside,” as a principal is likely to have a longer attention span if a trusted staff person has produced the document. Simple bullets with bolded key sentences or phrases make central points stand out. Finally, the piece should incorporate a straightforward narrative drawn from experiences in the field. A document that is too data-oriented runs the risk of becoming abstract and distant. A document designed to achieve these many different goals requires considerable effort and time, but the payoff is substantial.

To reach the general public, researchers and others must leverage the media via television, newspapers, or magazines. Because these pieces will be lengthier than the one- to two-page document for the targeted lawmaker, a narrative may be developed more completely to personalize the issue. Moreover, skilled writers or producers can use visuals to relay the key ideas quickly and memorably.

Today’s social media and online publishing options offer additional avenues. Increasing numbers of people, including policy experts, are now using Facebook, Twitter, and other social media applications to exchange information. However, our understanding of how social media can be used to promote policy change continues to evolve. For example, relatively little is known about what makes evidence go viral. A second challenge with these vehicles is credibility; virtually anybody can post information without regard to accuracy. That said, we are increasingly seeing information outlets in this space. For example, the Office of Policy Development and Research at HUD has an app that allows people to read accessible summaries of research and innovative practices in the field. Figuring out how to navigate these waters is a current frontier, and many learning opportunities are ahead.

For both the targeted and public strategy approaches, authors must take care to ensure that a narrative is not viewed as a one-shot exposé with limited generality or a thin advocacy piece. It must be clear that the evidence used to draw the conclusions is credible, definitive, and weighty. Sometimes, though rarely, a single study can accomplish this. However, a definitive conclusion regarding a particular policy issue typically arises through the cumulative effect of multiple studies conducted in varied contexts that produce a body of mutually reinforcing evidence.

The next section presents two case studies that demonstrate the power of narrative and vehicle for injecting evidence into policymaking. The first example describes a successful strategy that targeted the public. The second example shows the harm that can arise when an effective vehicle is absent. It also demonstrates how the subsequent introduction of an effective vehicle can change the tenor of the policy discussion. In both case studies, having a body of evidence that generated clear implications for policy was essential but not sufficient.

Evidence, Narrative, and Vehicle Working Together

Homelessness in Reno, Nevada: While working at HUD, I had a pair of “aha” moments during the budget-wrangling with Congress during President Obama’s first term. These moments revealed the importance of a narrative and a vehicle. During the 2010 and 2011 budget deliberations, Congress was in a serious belt-tightening mode. Line items were being pitted against each other to try to bring budgets in line with the reduced total spending that Congress authorized. At HUD, this meant difficult decisions regarding whether vouchers, public housing, block grants, or Secretarial initiatives should bear the bulk of the austerity burdens.

Homelessness was noticeably absent from the conversation about trade-offs. Almost nobody talked about reducing funding for the suite of programs designed to reduce the incidence and severity of homelessness in the United States. Why? Because everyone in Washington—from policy experts, to staffers on the Hill, to elected officials—shared the same understanding about the large returns to up-front investments targeted at treating and preventing homelessness.

The question, of course, is: How did such a consensus emerge? A key part of the answer can be found in an article by Malcolm Gladwell that appeared in The New Yorker in 2006, titled “Million-Dollar Murray.” The article tells the story of Murray Barr, a chronically homeless man in Reno, Nevada, and the police officers who were regularly called to pick up Murray and deliver him to the hospital or county jail. Gladwell reports that local police estimated that Murray had racked up at least $100,000 in hospital bills in only six months. But he’d been repeating the same pattern during his 10 years on the streets, meaning that he’d likely cost public services more than $1 million—far more than what it would have cost to provide him housing or supportive services.

Although the story would have been quite useful for informing homelessness policy in Reno, Gladwell went further. He chronicled the work of many researchers—including Dennis Culhane, now widely recognized as a leading researcher on homelessness—to highlight consistent evidence supporting the notion that there are Murray Barrs in every U.S. city. The key takeaway from Gladwell’s piece is that most of the costs associated with homelessness owe to the troubles of a small number of people who are chronically homeless. If we focus treatment on these people, he argues, we can see both short- and long-term savings.

While Gladwell provided the narrative (and it is good reading!), success occurred in part because The New Yorker was an ideal vehicle. Its readership is broad and it is popular among the better-educated urban people who would know of homelessness but not necessarily understand it. It gave this group, which undoubtedly included some policymakers and aides, a new way of thinking about the problem and its potential solutions. The vehicle helped the story quickly make its way through a broad set of circles and its takeaways became generally known. Hence, a common understanding emerged, and the funding for homelessness prevention policy was resilient in the face of intense budget stress. Evidence coupled with a narrative and a high-quality vehicle translated into the effective use of evidence to inform policy.

Housing Counseling: A second example is housing counseling. During budget tightening, unlike the case for homeless services, housing counseling became a target of congressional appropriators, and its line item was ultimately zeroed in the House’s budget prescription. Many assumed this was an impossibility. By 2011, everyone was aware of the housing crisis and its devastating effects on families across the country. It was widely understood that many people got into trouble due to a lack of understanding of the risks associated with some mortgage instruments and homeownership more generally. Moreover, stories abounded of how a specific housing counselor had saved the day for a desperate homeowner. So how could counseling be stripped of funding?

There were two issues here. First, somewhat surprisingly, policymakers did not know about the quantified benefits of housing counseling. After the House action, HUD convened the major housing counseling agencies and advocates to determine a response. During these discussions, it became clear that little effort had gone into building the case for the return on investment (ROI) for housing counseling. Counseling was at a disadvantage compared with other policy areas that had such cost/benefit figures because it was difficult to demonstrate that counseling was more cost-effective than another policy. Second, the lack of a vehicle was a problem. Appropriations staffers were unaware of the narratives regarding the benefits of counseling and did not hear about them during the budget process. Clearly, the narratives were not effectively deployed. These factors doomed the program.

However, the story ended on a somewhat happier note. The loss of funding galvanized the counseling industry to correct both of these problems. A coalition of key players, including service providers, advocates, and government staff, worked together to assemble data on the costs and benefits of counseling. Their analysis showed that housing counseling resulted in almost $400 in benefits for every $1 spent.[8] Second, a bevy of counseling providers and housing policy advocates descended upon the Hill with a coordinated information campaign to make sure that the cost-effectiveness and efficiency arguments were too loud to ignore. Ultimately, some funding for counseling was restored.

Advancing the use of Evidence to Inform Policy

Producing the narrative and finding the right vehicle consistently and effectively require drawing from the knowledge and expertise of people with varied backgrounds. Researchers and analysts are necessary to distill a body of research into its essential messages. Practitioners and advocates often are aware of the experiences that can put a personal face on the messages and provide the basis for a compelling narrative. Public affairs and media professionals are skilled in crafting pieces that have maximum impact. Purposeful collaboration among these groups will increase the likelihood of success. Yet few organizations set bringing together teams of people with these diverse skills as a key goal or mission. More need to. Governments can be effective in this regard, but changes in administrations bring changes in goals and priorities, so there can be an ebb and flow in government’s participation. Perhaps philanthropy, which can be more stable in its objectives, can play a catalyzing role in this regard.

A more fundamental issue is exemplified by the counseling example. The translation from anecdote to general lesson depends on the presence of indisputable evidence and consensus on what the evidence means. Success would be enhanced if there were monitoring to ensure that these preconditions exist and, if they do not, to determine what needs to be done to get the field to that position. In the case of counseling, the budget crisis would have been less likely to occur if there had been an organization that was mission-driven to ensure that there was a strong evidence base and that the lessons from existing evidence were easily available to key decision-makers.

This process would run something like this. Ask the question: Has anyone summarized what is known about a given policy? If so, the next issue is to wrestle with whether clear lessons or implications emerge from the existing evidence. It is likely that the author of any summary will make declarations about key takeaways. But independent scrutiny is important to preserve the credibility and validity of subsequent efforts. Furthermore, if no summary has been done, take the time to do it and craft the lessons to be drawn from what was found.

I am unaware of organizations that view this as their role. My unit at HUD embarked on a two-year Research Roadmap process to determine what questions in a number of policy directions still needed attention. The effort was difficult, precisely because few institutions and experts ask these questions in the course of their everyday work. This is a gap that requires filling, though there are some encouraging examples of this kind of work. The multi-institution What Works Collaborative, established at the start of the Obama administration, had as its goal identifying promising policy implications from existing research and supporting other research that had promise to answer key questions. Similarly, the recent collaboration between the Federal Reserve Bank of San Francisco and the Low Income Investment Fund that resulted in Investing in What Works for America’s Communities yielded a product that purposefully incorporates the varied expertise of multiple groups.

Along similar lines, we would benefit by having some institutions that viewed it as their charge to maintain a real-time, current record of what we know about a particular policy arena. Given the continuous flow of studies and reports issued by universities, think tanks, advocacy groups, and others on various issues, it can be a significant challenge to maintain pace with policy changes, and individual researchers often do not provide straightforward syntheses of their work that place the results in a useful policy context. Support to assign some individuals or organizations with this role would be quite helpful.

Finally, timing is a key barrier to the effective use of evidence in policymaking. Too often, evidence is not available when policymakers are looking for it. An ongoing, real-time summary of what is known would mean that answers to policy questions, such as we have them, will be readily available when the policymakers want the knowledge, rather than the present reality in which requests for knowledge are often met with a reply that study results will be ready in 18 months.

Conclusion

More evidence-based policymaking will require attention to all of the elements of the undertaking. We will need to compile the right data and conduct the highest-quality analyses and evaluations. As the housing counseling example demonstrates, assuming the appropriate evidence exists is problematic, even in mature policy areas. Housing counseling had been offered for decades, and yet the evidence was not there. We cannot advance effective policy if we do not know what works, or if we—at a minimum—can’t clearly demonstrate that certain policies work. Lacking an evidence base will almost always be fatal in this pursuit. We need to address evidence gaps, which will require an assessment of existing data and data systems to identify barriers and find solutions for overcoming them.

But we should also consider other parts of the equation, because the existence of evidence is insufficient to guarantee its use by policymakers. Is there a narrative that captures the essence of what we know in a way that is personal and memorable? We must find stories as compelling and straightforward as the Murray Barr story for all of our policy areas. We must then deploy resources and skills to ensure the story is told to maximize its effects. Do we have a vehicle that can spread this narrative broadly and to the right audiences, so that our knowledge becomes common knowledge, particularly among those involved in making policy? Too often, knowledge in academia never becomes general knowledge, or if it does, it happens several years after the initial point is established. The end results of slowly disseminated knowledge are more societal costs and fewer societal benefits than should be realized. Purposeful attention to identifying and leveraging the right vehicle to gain the broadest possible understanding of the evidence and its implications can significantly improve the likelihood that better policies are adopted and implemented.

Both these dimensions need to be someone’s responsibility. Ideally, an organization would take on this role so that institutional memory about the evolution of policy and evidence could be broadly shared. Without a “ring leader,” the use of evidence to inform policy will happen, at best, on an ad hoc or somewhat random manner. We can do better and should not leave such matters to dumb luck. Success here will mean having a robust strategy for effective use of evidence in policymaking, resulting in better policies that are adopted more quickly. Failure will risk having this conversation again, and again, and again. I, for one, would like the cycle to end here and now.

[1]   P. Orszag, “Building Rigorous Evidence to Drive Policy,” Office of Management and Budget blog, June 8, 2009. www.whitehouse.gov/omb/blog/09/06/08/BuildingRigorousEvidencetoDrivePolicy.

[2]   Orszag, 2009; Office of Management and Budget, “Circular Number A-11: Preparation, Submission, and Execution of the Budget” (Washington, DC: Office of Management and Budget, 2012).

[3]   Royal Society and National Academy of Sciences, “Climate Change: Evidence and Causes” (Washington, DC: National Academy of Sciences, 2014).

[4]   C. Liu, “MTA Sees Success in Orange Line,” Los Angeles Times, November 21, 2005, http://articles.latimes.com/2005/nov/21/local/me-orange21; A. Loukaitou-Sideris, D. Houston, and A. Bromberg, “Gold Line Corridor Study, Final Report.” (Los Angeles, CA: UCLA Ralph and Goldy Lewis Center for Regional Policy Studies, 2007).

[5]   J.W. Kingdon. Agendas, Alternatives, and Public Policies (Boston: Little, Brown, 1984).

[6]   RCTs present other challenges in a policy context. In some instances, particularly those that have cross-sectional elements, some may question whether the design includes sufficient controls to identify and disentangle causal effects. Moreover, RCTs often do not yield results in a timely manner.

[7]   G. Bel, X.Fageda, and M.E. Warner (2010), “Is private production of public services cheaper than public production? A meta-regression analysis of solid waste and water services,” Journal of Policy Analysis and Management, 29(3), pp. 553–577.

[8]   This estimate was generated from internal modeling and projections by the Office of Policy Development and Research at HUD.