FacebookTwitterEmail

Community development involves long-term change in a complex environment, with many actors, strategies, and variables. Keeping track of all these components, let alone evaluating success, can be difficult. However, “shared measurement”—a method for assessing and understanding complex change—has captured the imagination of the community development and evaluation fields as nonprofit organizations, intermediaries, and funders seek more effective ways to understand the change that their programs and investments are making in communities.

Many organizations support or deliver similar programs that have the same intended outcomes. For example, financial capability programs delivered by multiple organizations throughout the country all aim to build financial skills and behaviors. Other community development efforts engage multiple organizations, each delivering its particular piece of a comprehensive strategy to accomplish a set of common outcomes. This approach is common in collaborative community revitalization efforts involving multiple organizations or multifaceted youth development programs. In both cases, evaluation using shared measures can be an effective and efficient way for organizations to learn whether they are achieving their intended outcomes.

This essay explores the benefits and challenges of using shared measures in community development. It also illustrates how shared measurement can be used for larger scale evaluation efforts that more fully engage organizations and their funders in defining how best to measure results of programs, collaborative efforts, or the field of practice. Finally, the essay examines how shared measurement efforts have built capacity and vocabulary for measuring outcomes among practitioners, while reducing the time and effort spent by an organization on evaluation after a start-up investment phase.

What Is Shared Measurement? Why Does it Matter?

Shared measurement approaches engage multiple organizations in using the same indicators or data collection tools to evaluate the performance or outcomes of their programs, place-based strategies or collaborative initiatives. Although applied across the community development field in several ways, organizations typically use common indicators and data collection tools to evaluate similar programs implemented in different locations or for different populations. For example, organizations draw on sets of shared measures to evaluate the outcomes of programs such as affordable housing development or community engagement in communities across the county. They may choose to share measures to better assess the most effective means of delivering similar programs. Alternatively, organizations may be primarily interested in understanding the outcomes of their own programs, but want to enhance the validity of their own evaluation processes by drawing on tested measures used by other organizations in their fields. This use of shared measures can significantly streamline an organization’s evaluation design process, saving both time and resources. Finally, a funder or intermediary may ask grantees or affiliates to use certain measures in common. In this case, the funder or other evaluation sponsor typically requests that organizations share data with them in order to look across the participating organizations to better understand the nature and scale of the outcomes achieved. Often, funders or sponsors may share their analysis to enhance learning and enable organizations to compare their own results within the context of the aggregated data set.

The most frequently cited application of shared measurement is within collective impact strategies. In these efforts, using shared measures enables a group of organizations working collaboratively to evaluate their progress toward a set of mutually defined common outcomes. For example, shared outcome measures may be used to assess outcomes of a comprehensive set of youth development programs including education, health care, employment, drug prevention, and mentoring. Rather than looking at the results of each program separately, organizations use a common set of measures across the programs to assess outcomes such as high school graduation rates, postsecondary education success or training goals, and the ability of youth to assume leadership roles. Using common measures in this way, organizations are likely to better align their program strategies, learn from the evaluation process collectively and use the results to strategize system-level enhancements that can significantly improve outcomes. In the most effective applications of shared measurement within collective impact strategies, the evaluation process itself strengthens the collaboration and can be a catalyst in enhancing program delivery to achieve intended program goals.

An Evolving Practice

Starting in the early 1990s, community development stakeholders began addressing the need for standard measures by developing common definitions for performance measures such as housing units developed or rehabilitated, jobs created or commercial properties developed. One of the earliest efforts was the Community Development Financial Institutions (CDFI) Fund’s Common Data Project, notable for engaging a broad range of organizations within the emerging community development lending movement. Meanwhile, federal agencies were instituting performance measurement requirements, and national organizations, such as the International City/County Management Association, were developing a set of shared performance measures for local government services that covered response times for emergency services, provision of human services to youth and seniors, infrastructure improvements, and more.

As organizations began to master using common definitions to measure service provision and productivity, nonprofit organizations and funders wanted to better understand the broader change resulting from their work, including hard-to-measure changes such as quality of life and community resilience. Current shared measurement practice in U.S. community development emerged from this shift toward documenting outcomes rather than just units of service or other performance metrics.

Technology Fueled the Evolution

By the mid-1990s, advances in technology and greater availability of data led to the community indicator movement, another building block toward shared measurement.[1] The Urban Institute established the National Neighborhood Indicators Partnership (NNIP), which now includes 36 data intermediaries that support greater access to local and regional data on a range of community measures.

In the first decade of the 2000s, several sets of shared outcome measurement tools emerged. The earliest among these were the Aspen Institute’s FIELD program’s MicroTest for the microenterprise field; the U.S. Department of Health and Human Services’ shared measures for Individual Development Accounts; and the comprehensive set of outcome measures for community development programs developed by Success Measures, an evaluation resource group at NeighborWorks America.

Technology played a pivotal role in the growth of shared measures. New tools provided accessible and secure ways to collect, access, manage, and analyze information to evaluate programs and place-based change. These included Efforts to Outcomes (ETO) by Social Solutions; the Success Measures Data System (SMDS) by NeighborWorks America; Outcome Tracker by VistaShare; FamilyMetrics by Pangea Foundation; PolicyMap by The Reinvestment Fund; and the open-source Local Data application.

More recently, the impact investing movement developed sets of shared measures to monitor the financial, program, and social performance of a range of both domestic and international community development projects and social enterprises.[2] The Impact Reporting and Investment Standards (IRIS), an initiative of the Global Impact Investing Network (GIIN), is an example of a system designed to inform investors interested in tracking social return. The IRIS online catalog of generally accepted, shared performance metrics creates a common language for reporting social and environmental performance.

Today, the concept of sharing measures has become so widely accepted that, as a service to the field, online platforms such as PerformWell—a partnership of the Urban Institute, Child Trends, and Social Solutions—gather, categorize, and share data collection tools developed by other organizations throughout the country.

Shared Measurement in Practice

The following examples illustrate how organizations are using shared measurement strategies to track short- and long-term outcomes in community development programs and investments.

Using Shared Measures to Evaluate Similar Programs or Strategies

Individual organizations interested in using shared measures to evaluate programs such as neighborhood revitalization, financial coaching, community engagement, or small business development can draw on available sets of shared measures or join with others to define and develop data collection instruments. These organizations may share a funder or intermediary interested in looking across a portfolio of similar efforts. They may be joined together in a network committed to a set of principles and practices or be completely independent yet interested in using measures established and vetted by others in the field. Typically, these organizations are motivated to use shared measures because they are primarily interested in examining their own outcomes or fulfilling funder accountability requirements. Having the ability to compare their findings with those of other nonprofit organizations using the same measures may not be important to them or be of secondary interest.

Shared Financial Coaching Measures: A recent example of funder-sponsored use of shared measures is the evaluation component of the Financial Capability Demonstration Project, a partnership between NeighborWorks America and the Citi Foundation. Thirty nonprofit organizations in 17 states used a common set of measures to document outcomes of financial coaching services. Drawing on a set of shared measures developed with input, review, and testing by experts and practitioners across the asset building and financial capability fields, these organizations used one common survey instrument to document their clients’ household composition and measure changes in clients’ financial status and saving, debt, and credit behaviors.[3]

With the support of technical assistance providers to plan and implement an evaluation, each organization selected at least one additional data collection tool to capture outcomes related to the specific focus of its coaching program, such as banking access, budget management, or college savings strategies. Over the course of the project, organizations also benefited from two convenings designed to provide training and peer exchange on key evaluation and data use strategies, as well as best practices. Using the Success Measures Data System, or its own client management systems, each organization collected at least two rounds of data on a sample of its financial coaching clients.[4] This structure provided each organization the data it needed to understand its own results and guide program improvements. It also allowed the funder to assess outcomes across the 30 organizations.

Results were aggregated for measures such as:

  • Percentage of respondents who increased their credit score, as well as the mean increase.
  • Percentage of respondents who started saving for the first time or increased savings.
  • Percentage of respondents who decreased total unsecured debt and types of debt held, as well as the mean decrease in the amount owed.
  • Change in number and type of bank, credit union, and long-term accounts.
  • Changes in perceptions of ability to manage personal finances.

Shared Community Impact Measures: Another example of using shared measures for evaluation of similar programs is NeighborWorks America’s Community Impact Measurement Project, which involves 239 nonprofit members of the NeighborWorks Network.[5] These organizations are using the same measures to document change in communities across the country. The shared measures allow organizations to document the conditions of occupied and vacant residential and commercial properties, as well as levels of community engagement, resident satisfaction with neighborhood quality of life, and local economic impact.

In 2013, each organization identified an area of 500 to 1,200 households where it provides programs and received a data profile of the community’s demographic, economic, and housing characteristics. Using a set of surveys and observation checklists, each organization systematically collected data, drawing on staff and community volunteers to complete the fieldwork needed. To ensure quality in this primary level data collection effort, experienced evaluators helped organizations draw appropriate respondent samples, train data collectors, plan survey implementation, and ensure high response rates. Participating organizations will collect this data again in future years for comparison. Economic impact is also analyzed based on performance data reported annually to NeighborWorks America by each organization.

The organizations are using their baseline data for a variety of purposes. For example, a number of them found that there were a greater share of longtime renters in their communities than they had assumed and that many were interested in remaining in the community to purchase a home. These organizations used this finding to direct some of their neighborhood marketing efforts toward current renters. In other communities, a detailed inventory of property conditions revealed patterns of roof, porch or other minor repair needs that led directly to new programs to address these problems. Other organizations focused on data they gathered on residents’ confidence in their communities’ futures and residents’ willingness to become involved in working on community issues. In response, organizations are initiating or strengthening community outreach and engagement strategies.

Other noteworthy examples of this shared measurement model are Habitat for Humanity’s Neighborhood Revitalization Initiative and the Wells Fargo Regional Foundation’s evaluation of its neighborhood planning and neighborhood implementation grant programs. Approximately 100 of Habitat for Humanity‘s local affiliates throughout the country are using shared measures to evaluate the community impacts of their neighborhood revitalization programs. At the Wells Fargo Regional Foundation, based in Philadelphia, approximately 55 grantees working to revitalize communities in Delaware, New Jersey, and eastern Pennsylvania are assessing perceptions of community change using a common survey of residents’ satisfaction with a variety of quality-of-life factors.

Benefits

In addition to the more general benefits of integrating systematic evaluation into their programs, organizations report several gains in learning and evaluation by drawing on shared measures. Organizations report that using shared measures:

  • Helps streamline the evaluation design process.
  • Gives organizations a shared language and experience that fosters peer learning to improve service delivery or development strategies.
  • Allows organizations participating in citywide, regional, state, or national initiatives to contribute new understanding of program outcomes to the broader community development field.
  • The use of shared measures produced additional benefits for community development funders, researchers, and networks of nonprofits, including:
  • A set of longitudinal, quantitative and qualitative primary level data on the effectiveness of various approaches to financial coaching or place-based revitalization; these data can inform policy, programs, and revenue streams.
  • A shared evaluation vocabulary among funders and nonprofit organizations about outcomes.
  • Greater capacity among hundreds of participating organizations to plan and implement evaluation.
  • Standardized training and technical assistance processes that make it possible to implement an evaluation requirement across grantees or affiliates in ways that are perceived as fair and adequately supported.
  • New organizational capacity to collect and use data at the community level; this capacity enables community-based organizations to become more effective partners with researchers addressing broad research questions.

Challenges

Community development organizations, intermediaries, and funders cite several challenges in implementing shared measurement evaluations.

Community Development Organizations

  • Getting buy-in on the specific measures included in a common data tool. This is a particular challenge when programs target similar outcomes but employ different strategies, or when organizations are sharing data primarily with a funder or project sponsor but not with one another. This can be addressed by including in the common tool the core measures with broadest application and allowing organizations to add other data collection tools and questions to tailor their evaluation.
  • Allocating the staff or volunteer time required for collecting primary-level data directly from clients and community residents. Organizations address this challenge by developing partnerships, recruiting additional volunteers, or in the case of client data, helping staff more seamlessly integrate data collection into program delivery.
  • Addressing staff or leadership turnover and program or financial challenges. These organizational issues can easily derail a shared measurement effort regardless of an organization’s initial commitment. Building in flexible technical assistance to bring new staff up to speed is essential for keeping the evaluation effort on track if there is turnover. Organizations facing program design or funding issues frequently need to set aside their evaluation efforts until those matters are resolved. Flexibility on the part of funders, partners or technical assistance providers can allow organizations to successfully resume their evaluation efforts at a later time.
  • Ensuring data quality. This may be a particular challenge when an organization uses volunteers to collect data. A shared evaluation plan that clearly presents the data collection tools, data gathering process and method for checking data quality is essential. If also using a common technology tool for data collection and management, it is helpful to have a central source for accessible technical assistance to address issues organizations face along the way.

Intermediaries and Funders

  • Setting realistic funder or project sponsor expectations about the scope of the results. In many cases, owing to the differing variables in the programs, the evaluation is not designed to compare results of different program models. Rather, the aggregate analysis demonstrates general trends in client or community results.
  • Effectively directing technical assistance to support organizations in using shared measures. This is particularly challenging when many organizations are collecting data with the same measures. Online training, web-based materials, and remote phone and e-mail contact with a cadre of experienced evaluators on the topic have proven effective in addressing this issue.

Using Shared Measures to Evaluate Common Outcomes from Different Programs or Strategies

Shared measurement approaches are also used to evaluate collaboratively run initiatives striving toward collective impact. These efforts typically include a commitment among participating organizations to share data and, in some cases, jointly apply lessons learned. An example is the youthCONNECT initiative, a five-year partnership of Venture Philanthropy Partners and nonprofit organizations in the Washington, DC metropolitan area. YouthCONNECT is an effort to improve education, employment, and health behavior outcomes for low-income and at-risk youth, ages 14 to 24. Programs and services implemented by the six youthCONNECT network partners include college access; charter secondary education; youth development and services; HIV/AIDS prevention and treatment; and job and professional readiness through training, internships, and mentoring.[6]

When the youthCONNECT collaborative was created, the partner organizations committed to shared outcome measures to document progress. They defined the types of data each partner would collect and report. For example, organizations report which of their participants receiving youthCONNECT programming are enrolled in school and are on track to be promoted to the next grade level, which students graduate from high school, and which students enroll in and eventually complete postsecondary degrees. The partners also collect data on several other risk and protective factors, tracking data on youth who have positive adult relationships, avoid negative peer relationships, and avoid physical violence and substance use. Technical assistance provided by Child Trends played a key role in facilitating decisions about the evaluation design and is keeping the implementation of the shared measurement effort on track.

Benefits

The youthCONNECT partners identified key benefits of their shared measurement model. They found that it:

  • Reinforced partners’ commitment to collaborative efforts. Although the partners recognized that the toughest problems cannot be solved by any single funder, program, or agency, collective efforts were difficult to maintain. The shared outcome framework helped maintain accountability and participation by all engaged partners.
  • Enabled peer exchange and learning. The use of common measures promoted teamwork when addressing data challenges, sharing training tips, and in peer-to-peer consulting.
  • Advanced project goals and created opportunities for the organizations to collectively reflect on their programs. Discussion about common measurement and evaluation procedures pushed the partner organizations to consider program refinements and to become more sophisticated in their thinking about key definitions and categories central to understanding their impacts.

Challenges

Key challenges identified by youthCONNECT partners include:

  • Allocating the staff time and resources to the process. Developing and implementing a common outcome framework required significant time and commitment as well as a high level of critical thinking from each partner. This was managed by assigning one representative from each partner organization who could attend regular meetings, engage in organizational and community-level measurement discussions, and report to the respective organizations. However, the staff time still far exceeded initial estimates and, in response, the project funders provided supplemental funding to compensate for the additional resource needs.
  • Agreeing on shared measures. Each partner organization needed to relinquish some degree of organizational autonomy for the project to succeed. To that end, organizations dealt with some of the practical challenges of collaboration by adopting common terminology, building relationships among the organizations’ staff, accommodating program model diversity, and addressing differences in capacities to collect and use data.

Learning From Practice: Is Shared Measurement a Good Fit?

Shared measurement can be a useful, scalable approach that benefits individual organizations and advances practice on the whole. However, shared measurement takes time, resources, and sustained commitment. It also requires at least basic levels of data management and evaluation capacity within an organization and a willingness to balance individual organizational needs with the use of a standardized tool or framework. Funders and sponsors must have a realistic view of the cost to launch and sustain this effort. The time that nonprofit organizations and public-sector agencies must devote to a shared measurement effort must be adequately supported, particularly in the start-up phases. In addition, shared measurement cannot always be effectively aligned with proprietary program requirements in which each funder defines its own distinct outcomes, grant reporting requirements and technologies, and evaluation cycles.

How can organizations know whether shared measurement is appropriate?

Many factors must be considered, but the following questions can guide a range of stakeholders in determining whether using a shared measurement approach can advance their particular evaluation objectives.

For funders, intermediaries, public agencies, and other network or project sponsors:

  • Are you interested in looking at outcomes across a grant portfolio, multisite initiative, or other major multi-organization effort?
  • Is building the evaluation capacity of grantees, affiliates, or network members a priority?
  • Does your organization have an interest in enabling peer learning among grantees or affiliates?
  • Are you seeking tools to help organizations streamline outcome tracking and reporting?
  • Will you need to scale an evaluation effort to a larger number of organizations or locations?
  • Are you planning to sustain an evaluation effort in multiple sites through multiple rounds of analysis?

For local or regional nonprofit organizations:

  • Are you involved in collaborative projects or part of a network interested in understanding your results according to common measures?
  • Do you like the idea of having a head start in planning your evaluation process by drawing on sets of measures or data collection tools that others have developed for programs in your field?
  • Are you interested in building your organization’s ability to collect and analyze data rather than using an external evaluator?
  • Is collecting qualitative data directly from program recipients or other local people or places essential to measuring your intended outcomes?
  • Do you believe that shared measurement has the potential to advance learning in the field in a meaningful and consequential way?

If the answer is “yes” to several of these questions, shared measurement approaches can be a good fit. Organizations will also want to consider the range of practices, tools, and incentives that can make shared measurement more effective and useful. Ensuring adequate support and incentives can tip the scales to help sustain commitment to shared measurement by reducing the burden for nonprofits and funders. The following ideas drawn from applications of shared measurement provide a good starting place for organizations new to this approach.

Practices

  • Providing training and technical assistance for practitioners is critical to ensure the collection of quality data and to troubleshoot organizational barriers to instituting shared measurement processes. Accessible, online training to build capacity is a scalable option, providing an affordable way to effectively reach a large number of organizations.
  • Convening peer-learning opportunities, both virtual and in person, strengthens application of data to everyday organizational uses. Having a forum to share challenges and best practices can be valuable for practitioners and funders seeking to better understand the impacts for their investments in people and communities.
  • Sharing successful examples of shared measurement models has the potential to advance community development and evaluation practice.

Tools

  • Providing data collection tools that can be used in common is critical to successful shared measurement. To take full advantage of current and ever-evolving quality tools, investments must be made in menus and libraries of tested and relevant data collection instruments. Although the initial time and investment required can seem prohibitive, the resulting products form the backbone of a shared measurement process.
  • Using technology to share measures and support data management, analysis, interpretation, and reporting is essential. Investment in technology can make a critical difference in the ability to consistently collect and aggregate quality data in shared measurement efforts.

Incentives

  • Supporting a streamlined evaluation planning process through shared measurement offers a clear, practical way to determine what outcomes to measure and how to measure them. Organizations save time and planning effort because they do not need expertise in all aspects of evaluation to achieve quality evaluation design, tools, and analysis. Menus of common measures and data definitions that are easily accessible and understood by practitioners are a vital building block of an effective shared measurement process.
  • Providing easy access to secondary data on communities, including key demographic, social, economic, education, housing market, human service, public safety, and other factors, is critical to shared measurement efforts in community development. Support for data intermediaries to assemble, analyze, and disseminate this data makes it possible for nonprofit organizations to cost-efficiently share data from secondary sources, streamline data access, and lower costs when primary data are not needed to address their evaluation questions.
  • Adequately funding the evaluation activities to build and sustain shared measurement efforts is essential to recognize the value of an organization’s time and to provide sufficient support for data collection, technology, analysis, peer engagement, or other specific needs. Clarity and commitment of funder support for shared measurement efforts are powerful incentives.

As this overview of shared measurement makes clear, application of the practices, tools, and incentives helps advance the field’s understanding of the impacts of a range of community development programs and investments. As it continues to gain traction, this rigorous yet flexible method for capturing complex change is becoming an integral part of the community development tool kit, increasing the scale at which evaluation of people and place-based initiatives occur, as well as fostering stronger, more vibrant communities in the process.

Documenting Health Outcomes of Housing and Community Development Programs

As more community development practitioners embrace the important linkages between community development and individual, family, and community health, the need for tools to measure these health-related outcomes has become increasingly clear. In response, Success Measures® based at NeighborWorks® America, is developing an evaluation framework and set of data collection tools for community development practitioners interested in documenting the health outcomes of a wide range of affordable housing, neighborhood revitalization, workforce development, supportive service, and community engagement programs. Drawing on social determinants of health research and using a health equity lens, the project is focused on the evaluation support needs of community development practitioners and complements efforts by leading funders, researchers, and others to identify core measurement issues at the intersection of health and community development.

Based on a literature review, stakeholder engagement, and the input of advisors drawn from the health care, public health, community development, and public policy fields, the final products of this effort will include an evaluation framework arraying the health outcomes along the spectrum of housing and community development programs, and a set of tested, validated data collection instruments to measure those outcomes. Similar to the data collection tools currently available through Success Measures and the Success Measures Data System (SMDS), these instruments will be developed for a range of community, cultural, and program settings, including translation into several languages, and will be applicable across populations from youth to seniors. Completion of the evaluation framework is anticipated in early 2015, with the data collection tools following later in the year. For periodic updates on this project, including information about opportunities to collaborate and participate in field tests, please see www.successmeasures.org.

[1]   For more information on the community indicator movement, see Ben Warner’s essay in this volume.

[2]   J. Freireich and K. Fulton, “An Industry Emerges.” In Investing for Social & Environmental Impact. (San Francisco: Monitor Institute, 2009).

[3]   NeighborWorks America, “Measuring Outcomes of Financial Capability Programs: Success Measures Tools for Practitioners.” (Washington, DC: NeighborWorks America, 2011).

[4]   NeighborWorks America, “Scaling Financial Coaching: Critical Lessons and Effective Practices.” (Washington, DC: NeighborWorks America, 2013). www.nw.org/FinCoaching13.

[5]   NeighborWorks America supports the national NeighborWorks network of 245 independent community-based nonprofit organizations serving more than 4,600 communities nationwide.

[6]   The youthCONNECT partners include College Summit-National Capital Region, KIPP DC, Latin American Youth Center, Metro TeenAIDS, Urban Alliance, and Year Up-National Capital Region.