FacebookTwitterEmail

Are we making a difference?  What impact do the efforts of community developers, public health officials, social service providers, and others have on the lives of children and families? The answers to these seemingly simple questions are frustratingly elusive. In part, this is because our ambitions—to make sure that the places where people live provide opportunities for all children to grow up healthy and ready to learn, and support economic stability and advancement for all residents—entail complex interactions among people, institutions, and policies.  As such, getting a true grip on what is working, for whom, under which conditions, and compared to what, is far more complicated than counting numbers of houses built or programs financed. To fully comprehend the reach and impact of our efforts, we must align information from across sectors and institutions working to improve communities. And that is not easy.

The authors in this volume contend that developing better data “infrastructure”— which includes technologies, but also skills, capacities, and leadership—can help. Improving how we collect, and more critically, share data across institutions, they argue, can advance our efforts to see the connections across sectors and better understand the outcomes of our work. By linking the data of a public health clinic with attendance records at schools, or aligning housing data with information on transportation networks, for example, we can more easily spot resource gaps or opportunities for amplification of positive outcomes. This, in turn, can help us work together in joint action across sectors. Ultimately, better data infrastructure can help us be more strategic and efficient in our work, and can yield the information needed to influence policy and decisions about resource allocations.

As Kathryn L. S. Pettit and G.Thomas Kinsgley noted at the outset of this volume, the community development field—broadly defined to include all who work to advance equitable outcomes for low-income communities—is at an exciting moment. We now have more data than ever, and we have the computing power to crunch it. The authors in this volume point to measurement tools, indicator systems, and visualization strategies that exponentially increase our ability to galvanize a wide array of community actors, from residents to service providers to funders to politicians, to challenge current ways of doing business, and to develop novel strategies for intervention design and implementation.

But they also point to the progress yet to be made.  They outline the practices that must change to ensure that the data we collect are as representative as possible of community conditions, and that we are using it in ways that reflect community interests.   They draw attention to the range of capacities that must be built among funders, leaders, and front-line service providers to use data meaningfully.  They discuss the need to boost data transparency and to further connect the dots among various types of data to better assess the performance of programs and policies relative to diverse social and economic factors. Although it is impossible to fully capture the richness of the authors’ insights, what follows is a synthesis of key themes in the volume.

Improving Raw Materials

It’s a basic point to make, but our tools and systems for making meaning out of data are only as good as the data they harness. Although there will always be gaps, errors, and omissions in any data set, the authors point to a number of key considerations in ensuring that data assembly practices are capturing the complexity of the communities we serve as accurately and fully as we can.  The first is nuance. The data that we often count on to gauge progress— poverty rates, unemployment rates, educational attainment rates—are blunt instruments at best. We must dig deeper, in both collecting and verifying data, if we are to gain a thorough understanding of which programs and policies work to effect positive change.

One point that the authors stress is the importance of routinely double-checking data during collection to ensure its validity.  For example, Ira Goldstein discusses the importance of “ground-truthing” data to ensure that data accurately reflect on-the-ground conditions. Validation is especially critical when data systems pull information from secondary data sets that may be flawed or outdated, or when there is some degree of automation in data assembly.  Goldstein discusses how, in the case of the Market Value Analysis (MVA), which aggregates data from a variety of administrative sources, a built-in data validation process uncovers, and corrects, flaws early, before conclusions are drawn. Annie Donovan and Rick Jacobus, who discuss data systems that automatically consolidate and standardize transaction information, point out that those who are closest to real-world conditions should review the first run of charts and graphs to ensure that the data reflect reality and that analysts are drawing conclusions that make sense.

The authors also stress that we need to pay attention to data comprehensiveness, and the need to assemble data from multiple sources to decipher trends driven by multiple factors.  In their discussion of the National Mortgage DataBase (NMDB), Robert Avery, Marsha Courchane, and Peter Zorn discuss how none of the current data sources about the mortgage market include all the data necessary for accurately benchmarking, tracking, and evaluating the marketplace.  By bringing together myriad data, the NMDB will enable a deeper understanding of market dynamics.  In a similar vein, Claudia Coulton discusses how difficult it can be—and offers solutions—to accurately gauge residential mobility and resulting neighborhood change using only one data source. Several authors discuss how community residents themselves are important sources of data and information, and make the case that qualitative data can help sort out the “why” and “how” of community change. Meredith Minkler advocates for community involvement in collecting and analyzing data, and shows how community-based participatory research can enhance data validity and yield unanticipated insights that can ultimately improve program and policy design. In a similar vein, Patricia Bowie and Moira Inkelas discuss how surveying  residents about their experiences with neighborhood services and programs can help front-line providers and program leaders more fully assess whether programs are ultimately delivering on outcomes they set out to achieve.

Our authors also stress that timely data are critical, particularly if our aim is to be responsive to dynamic community conditions. Bowie and Inkelas discuss how the Magnolia Place data dashboard synthesizes monthly survey data in “real-time” so community workers can assess their ongoing roles in community change and adjust practices to better achieve goals. Likewise, David Fleming, Hilary Karasz, and Kirsten Wysen call for greater use of intermediate outcomes and rapid-cycle surveys on health and community development outcomes to more accurately determine whether mid-course corrections are needed.  Ira Goldstein also underscores the point that frequently updated data are vital for understanding neighborhood dynamics, and notes that the local government administrative data in the MVA, which are more frequently updated than sources like the census, better capture how market conditions change over time.  The key in all these efforts is the need for ongoing access to detailed data and trends so practitioners and leaders can quickly learn about changing conditions and adjust accordingly.

Making Data Meaningful Entails More than Collecting It

Many of the authors reinforce the point that data do not do anything on their own; it takes skill to make the information meaningful. While technical skills for manipulating data tables are important,  we need to build a wide range of “softer” skills to use data effectively and appropriately on a long-term basis.

Fundamentally, to make data actionable,  it must be easily digestible by a range of audiences. Maps and infographics can often help make complex information more accessible. Bridget Catlin describes how data visualization tools used by the County Health Rankings have evolved to help laypeople understand the multiple factors that contribute to health outcomes and to help users with different skill levels access data in formats that are meaningful to them. Storytelling and marketing skills are no less important in bringing data to life. Raphael Bostic hammers home the point that without a compelling storyline and a method to reach key audiences, decision-makers will not fully absorb or use data and evidence, regardless of their quality and import. He proposes a central repository where data and evidence on similar topic areas can be distilled and readily accessed by everyone involved in data-driven decision-making. Catlin also speaks to this issue, noting that for data and evidence to ultimately influence policy and program decisions, findings must be packaged for the media. Staff at the County Health Rankings, she says, work closely with communications experts to develop messages that can be widely disseminated to multiple audiences.

Many other “translational” skills are needed to help practitioners inject data into decision-making.  Victor Rubin and Michael McAfee discuss how, in PolicyLink’s role as a technical assistance provider, they helped Promise Neighborhoods and Sustainable Communities grantees to understand how to use data to guide decisions and manage program performance and community relationships. Alex Karner, Jonathan London, Dana Rowangould, and Catherine Garoupa White similarly discuss how the Center for Regional Change at the University of California, Davis, supplemented its social equity mapping tools by helping groups build their capacities in navigating the political landscape and framing controversial issues. These skills have helped community partners use the data to align policy and program changes across fragmented jurisdictions.

Another key idea that runs through many of the essays in this volume is that, above and beyond technologies and staff capacities, organizational culture and leadership approach can significantly impact the quality and sustainability of data collection and use.  These factors relate to the priority that is placed on data collection, analysis, and use, which in turn affects how time, money, and attention are allocated to these processes.  In other words, establishing data-driven practices, whether to measure impact, adjust programs, or foster accountability among partners, takes commitment not just from data analysts, as Susana Vasquez shows, but also from funders, leadership, and other staff. Echoing the point on the importance of funder commitment to this work, Alaina Harkness calls attention to the need for foundations to boost support for expansion of data capacities among grantees. Erika Poethig speaks to how these issues play out in local government settings.  She describes how strong data practices became embedded into management processes within the City of Chicago’s Department of Housing, where it took strong internal leadership to implement and sustain improvements in data collection and use.  Cory Fleming and Randall Reid similarly discuss how in Baltimore, sustained leadership and accountability mechanisms prompted local government agencies to align around using performance assessment tools in designing program strategies and adjusting practices as needed.

Finally, the authors remind us that data are not neutral. Rather, there are values and assumptions that underlie what and who gets counted, and how much one data point counts versus another.  As such, we must hone our critical thinking skills and be alert to the biases embedded in our data collection and measurement frameworks.  A number of authors discuss, for instance, the considerable subjectivity built into seemingly objective data platforms and systems, and how the ways that we define what counts shapes, and in some cases, limits, program and policy design.  In each of their essays, J. Benjamin Warner, Bridget Catlin, and Ira Goldstein point out that analysts and planners must make an array of choices when synthesizing and distilling data into more manageable and actionable information. These include decisions about which data to assemble and analyze in the first place, which benchmarks to use in assessing progress or change, and the thresholds that separate success from failure.  Warner explicitly considers how different frameworks and political orientations affect the scope of data that is used for assessing community conditions and the range of interests that are taken into account in decision-making processes. Several authors underscore that just the process of defining problems and solutions can be fraught.  Bowie and Inkelas  note how the scope of a logic model identifying causes for particular problem, if too narrow, limits the range of data that is collected and the questions that are asked of that data, ultimately shrinking the solution set.  In a similar vein, Ian Galloway considers the ways that impact investors choose to define “impact,” and how investor preferences might skew resource allocation.  All of these factors, which ultimately depend on human decisions more than data, can significantly affect the design and targeting of data-driven programs and policies.  As such, we need to keep our eyes open and take corrective actions when certain populations or practices are overlooked, undercounted, or misrepresented by the data feeding our decisions.

But What Exactly Are We Supposed to Measure?

One cannot make data actionable without first collecting it. For many communities, this basic question is the hardest: what data should we collect and analyze? Alas, there is no silver bullet. Instead, the authors raise a number of considerations that can inform how we craft data systems to help make sense of the context of our work and to ascertain the real impact of the investments and programs we implement.

Several authors, for instance, speak to the merits of data standardization. When multiple organizations use shared metrics, they argue, it’s easier to compare performance across organizations, initiatives, and communities.  Paige Chapel discusses the importance of standard metrics for community development financial institutions (CDFIs), arguing that uniformity is critical for accessing capital and investments.  Annie Donovan and Rick Jacobus show how standard data aggregated across organizations allows groups to compare performance and identify market conditions that all organizations may be facing. David Fleming, Hilary Karasz, and Kirsten Wysen call for uniformity and standardization in health data, which would allow for a quicker grasp of which health interventions work in various settings and the cost-effectiveness of implementing different types of approaches. Maggie Grieve speaks to the strengths of shared and standardized metrics in helping to align the actions of multiple organizations working toward the same essential goals.

A challenge with standardization is that, by definition, it does not accommodate nuance.   Bridget Catlin and Ben Warner each raise this issue in discussing rankings and indices.  These types of tools can help distill disparate data into standardized metrics that can grab headlines, and thus help to direct attention to important issues. But the standardization and simplification means a necessary loss of information. As a result, disparities can be masked, and conditions that are unique to particular communities or populations can be overlooked. More detailed and granular information is typically required to sort out how to design and target programs that meet diverse community needs.

Adding to this point, Bowie and Inkelas note that different actors—from community residents to front-line providers to managers to policymakers—need different types of information during design, implementation, and assessment of community interventions. Therefore, data collection and reporting systems must be expansive and nimble enough to accommodate different types of data, such as data on individual actions, program performance, and community context.  Alaina Harkness makes a similar argument, suggesting that a data system that aims to capture the complexity of community conditions must accommodate:

  • Quantitative and qualitative measures of change;
  • Both inputs (dollars in, resources on the task) and outputs (community gardens planted, murals painted, units of housing developed, people assisted);
  • Contextual data—real-time information about a neighborhood and individuals;
  • Process or “platform” data—information about how social and organizational networks are growing, and the added value of increased organization and connectivity.

Of course, the idea that every organizations data system must “do it all” is overwhelming. Essentially, though, the authors are suggesting that some set of common standards and definitions is vital for comparing performance of programs, aligning activities among organizations working toward similar ends, and understanding large-scale trends.  At the same time, we’ll need to be able to weave together additional layers of data to understand the context and reach of our activities, and the process of change, across geographies and populations.

The former will most likely require organizations to relinquish some degree of control and power over decision-making, a requirement that often doesn’t come naturally, to say the least. Maggie Grieve, for example, suggests that organizations participating in a shared measurement framework have to give up some degree of organizational autonomy and learn to accommodate the approaches of partner organizations to data collection and definition.  Similarly, Bill Kelly and Fred Karnas suggest that efforts to create consistent data across organizations will require leaders to step back from “proprietary” processes, some of which likely entailed significant expense and time to establish, and consider alternatives.

The latter will necessitate enhanced access to and interoperability among data sets, regardless of whether data are being generated by a CDFI, a health department, or a school.  Aligning data from various institutions and agencies operating in a given area or serving a given population will be critical for making sense of how particular programs and investments play out in concert with the full complement of community- and regional-level activities also in play.  To make a more significant leap forward in understanding how to achieve holistic and lasting impact, public, private and non-profit institutions will need to consider ways to make individual stores of data more commonly available to others.  The next section looks at this topic in more depth.

Boosting Data Access and Connecting the Dots

We place many demands on data; we seek a mirror and a map, odometer and oracle.  To achieve these ends, community-serving entities must more broadly and consistently increase data transparency and link fragmented data sets together.  The authors point to many different types of data sets that can be linked:  individual behavioral data, community or regional socioeconomic data, organizational performance data, government administrative data, and private, corporate data.

One problem in putting these data sets to use, though, is that many are locked behind organization walls.  However, in certain corners, this is changing.  Emily Shaw discusses the growth of “open data,” noting that government entities are increasingly publishing administrative data for use by the public as a way to support good governance and community engagement in planning decisions.  Ren Essene, along with Robert Avery, Marsha Courchane, and Peter Zorn, discuss how increased access to mortgage data can better identify market trends and performance in fair lending practices.  Eric Bakken and David Kindig discuss how greater access to data on how hospitals are allocating community benefit resources can help promote coordinated action among multiple groups tackling the social determinants of health.  Amias Gerety and Sophie Raseman discuss My Data, a system that allows individuals to access their personal data from government or corporate sources and share it with third parties.  They point to the possibilities of using both open data and My Data to build smart disclosure practices, where previously unavailable data can be aggregated in safe and secure ways to help consumers make better decisions and push suppliers of goods and services to change their practices.

Data access is only one piece of the puzzle, though.  The authors suggest that a more critical component of this work is about directly or indirectly linking data across domains, for example, across housing, health, and education programs. Linking data can help us better understand and address the interwoven nature of the factors affecting individual and population outcomes.  Encouragingly, they point to a number of tools and approaches that hold promise in this effort.  Aaron Wernham, for instance, points out that Health Impact Assessments, which assemble multiple types of quantitative and qualitative data about housing, transportation, and the environment, can help urban planners and developers understand how building designs, redevelopment plans, and transit infrastructure can be recalibrated to improve health outcomes. Nancy Andrews and Dan Rinzler discuss how a “social impact calculator” can use social science research to estimate the impact of affordable housing or early childhood education on health improvements and lifetime earnings.

Other authors discuss data approaches that trace how individuals interact with multiple programs and institutions.  Rebecca London and Milbrey McLaughlin discuss the use of Integrated Data Systems (IDS), which link individual data from across agencies. These systems help combat the gaps and redundancies in service that spring from institutional isolation. They speak in particular to the possibilities for IDS to better support youth. The linked data from multiple social service institutions can allow them to craft interventions that connect the dots among youths’ social, cognitive, emotional, and physical development.  John Petrila similarly points out that IDS can help better diagnose problems and develop solutions by identifying how interwoven systems – from health to environment to housing – affect individuals and communities over time. Bill Kelly and Fred Karnas tackle the topic of data linkage from a different angle, noting that  the Outcomes Initiative prompts affordable housing developers to collect and use data on health, employment, education, and other critical facets of resident life to uncover how service integration works and to spur policy development that supports the “whole person.”

Boosting data transparency and linking across data sets that include personally identifiable information raise important questions about data privacy and confidentiality. Rebecca London and Milbrey McLaughlin consider the multiple facets of this issue.  Organizations that agree to share data with one another must establish trusting relationships and develop agreements about how the data will be used and accessed.  They also must determine how to share and store data such that all parties are compliant with relevant data safety and security protocols.  Petrila cites resources that offer guidance on crafting these kinds of data use agreements, and considers some of the technological and methodological advances in storing data and matching individual data across data sets that offer enhanced privacy protections.

Moving Forward

It is likely apparent by now that this volume is only partially about data in and of itself.  The authors ultimately focused less on metrics than on the cultural and institutional factors that enable or impede efforts to learn about what is working, or what might work better, to collectively generate positive outcomes for the communities we care about most.

In part, this is because we are at the front of a new era in the work of creating opportunities in low-income communities.  Community developers are beginning to work differently, employing new strategies that simultaneously engage adjacent sectors, including health, education, public safety, transportation, and others. We are making strides toward what Paul Grogan called out in our prior volume of essays, Investing in What Works for America’s Communities:

What then is the future of community development? It lies in turning the architecture of community development to meet urgent challenges of human development. How to turn a successful community organizing and real estate development system toward the goal of increasing educational outcomes, employment success, family asset building, individual and community resilience to weather setbacks? As an industry, we need new strategies to face these challenges.[i]

The cornerstone of the new era of community strategy is data.  It is the tool that allows us to set a baseline; provides us the language to speak to one another across sectors and align our work; gives us feedback so we can constantly fine tune our interventions to meet the evolving needs of communities; and ultimately helps us to communicate “wins” to the communities we serve and to government, foundations, and other funders and investors.

Our authors tell us that being data-driven goes far beyond brandishing skills with calculators and rulers.  Rather, they send home the message that communities are complex systems, and that our approaches to understanding the part we play in community change must match the complexity at hand.  There is no single answer for how to do this.  But at a minimum, we need to embrace the idea that data is for learning.  Too often, data collection and analysis are seen as extraneous to the “real work,” important only for reporting or compliance procedures—for checking boxes and passing the test.  But under a learning mindset, data become key for understanding performance, improving practice, and holding ourselves accountable to our real clients, the communities we serve.  A learning mindset motivates data- and information-sharing that can yield collective knowledge and action.

But that’s just the beginning.  For new practices to take root, substantial investment will be required.  The more we invest in capacities to collect and align data, the better we will be at learning what works and what doesn’t in supporting families and communities. In turn, this will help us use resources more wisely and generate better outcomes. This volume intentionally brings together ideas from practitioners, researchers, funders, and policy experts from multiple fields to begin the conversation about what will need to change, from organizational behaviors to funding patterns to regulations and accountability mechanisms, if we are to better assess and articulate the impact of our work in low-income communities. As did Investing in What Works for America’s Communities, we hope this volume serves as a model for continued cross-sector dialogue and action to help achieve what should not be such a lofty goal: a nation where everyone has a fair shot at living a healthy, fulfilled life.

The views expressed in this essay belong to the author and do not necessarily represent the views of the Federal Reserve Bank of San Francisco or the Federal Reserve System.

[i] Paul Grogan, “The Future of Community Development.” in Investing in What Works for America’s Communities, edited by the Federal Reserve Bank of San Francisco and the Low Income Investment Fund. (San Francisco, CA: 2012).