FacebookTwitterEmail

Many cross-sector and multidisciplinary efforts are underway, all aimed at achieving better outcomes for residents within a community, neighborhood, or other designated area. Whether known as collective impact, “Promise” or “Choice” neighborhoods, neighborhood revitalization, cradle to career, place-based, or comprehensive community initiatives, these efforts seek to improve outcomes not only for a specific group of individuals or families, but for the full population across the life course. Although this is a time of optimism for the newest attempts in this arena, it is also an important time for reflection. What works, what doesn’t, and why?

A major challenge in answering these questions is that measuring change and impact is exceedingly difficult in cross-sector, geographic-focused initiatives. In large part, this is because we have not adjusted our practices to fully acknowledge that communities operate as “complex adaptive systems.” In complex adaptive systems, linear relationships between cause and effect do not apply. Instead, change is driven by constant and, frequently, unpredictable interactions among all elements of the system. In communities, this plays out as intentional, or unintentional, interactions among a range of community stakeholders, including residents, those providing services and supports, and others who shape community resources from both inside and outside the community. The dynamism inherent to the interactions among all these players renders standard models of organizational practice and leadership, and in particular, measurement, incomplete.

Through our long-term involvement with the Magnolia Community Initiative, and more recently with other community initiatives striving to improve population outcomes, we have had the opportunity to rethink the approach to measuring change under conditions of complexity. In this essay, we illustrate a set of measurement principles and approaches that have helped service providers and community residents better understand and respond to the complexity inherent in community change.

Background

The Magnolia Community Initiative is a voluntary network of 70 organizations and resident groups working in a five-square mile area near downtown Los Angeles. The network’s vision is to help the 35,000 children within the 500-block area break all records of success in their education, health, and the quality of nurturing care and economic stability they receive from their families and community. Partners include multiple departments operated by the Los Angeles County Chief Executive Office, including social services, child support, and child protection. They also include regional organizations responsible for populations of children such as the Los Angeles County Unified School District; the Women, Infants, and Children (WIC) nutrition program; and child care resource and referral services. In addition are the private and nonprofit community-based organizations providing health care, early care and education including Head Start and Early Head Start, family support, and banking and economic development services and supports.

The Magnolia Community Initiative aims to connect these diverse groups, programs, and providers in a system of shared accountability, emphasizing sustainable and scalable data-driven practices. This involves creating a systemic approach to community change based in problem solving, learning, and discovery. Data are employed to help vast numbers of individuals—including residents, providers, and leaders—identify ways to change their behavior to collectively improve conditions and outcomes for a local population of children and families.

Change in a Complex System

Approaches to community change have typically operated under an assumption that there is a linear order and predictability to achieving positive results. Standard strategic planning starts with defining the problem, deciding on a strategy or action plan, establishing a timeline with benchmarks to mark progress, implementing the interventions, and then confirming through measurement and evaluation that the intended results are achieved. The result is a blueprint of success that others can follow.

However, if we begin by acknowledging communities are complex adaptive systems, it is easy to see that the constant interplay of actors and actions produces outcomes that are impossible to fully predict, and that “success” may look different from what we imagined. In systems, change is constant, and there are too many moving parts to plan for every circumstance. As complex systems, communities are never permanently “fixed.” Although programs can be planned and implemented, it is impossible to plan and specify all of the detailed actions necessary to produce a better community-wide or population outcome. As such, improving outcomes for a community goes beyond the “right” service strategy, resource, or planning process. Similarly, measuring progress and evaluating success in these endeavors must go beyond tracking the “right” set of high-level benchmarks or indicators.

Complexity expert Brenda Zimmerman offers a grounded way of thinking about the unique approaches required under conditions of complexity.[1] She uses the examples of baking a cake, sending a rocket to the moon, and raising a child to distinguish between simple, complicated, and complex conditions, respectively. For the first two, one can be reasonably assured that with the right level of knowledge and expertise, the results will be similar each time. However, with complex problems, like raising a child, having expertise or experience offers no certainty of success. Any parent knows that strategies used to raise one child may not apply to a second child. In complex tasks, we must continually seek information that helps gauge whether the theory undergirding actions is relevant to real-life conditions. If not, we must adapt our actions and seek alternative solutions.

Data Can Provide a More Complete Picture  of Change

Gathering data on multiple levels across the system is critical for generating a more complete picture of what is happening and how the system is performing. To understand baseline conditions, the Magnolia Community Initiative developed a system map, which allowed the network partners to examine how the various actions of service providers, residents, and others were linked in affecting community results. In looking across the system and coming to a shared understanding of the interdependencies across players and chain-reactions among their actions, network partners could begin to see avenues for changing patterns in the system that would produce better results.

The Magnolia Initiative partners then instituted techniques to gather and display timely data and information about multiple components of the system. Our aim was to learn how the system was functioning in (nearly) real time to refine and continually improve actions. Timely data are critical to this approach and allow us to fine-tune changes at a much faster pace than what is typically possible from program evaluations or research.

Figure 1a. The Dashboard monitors resident experiences and trends in the community overall and in particular Magnolia Initiative partner settings. This chart shows a sample of a Care Process measure, which assesses the extent to which organizations are carrying out the work that the Initiative partners believe will improve outcomes. Goals (dotted green line) are set at ambitious levels that are considered achievable. Data are self-report measures from monthly client surveys collected by organizations. The Dashboard also includes charts that monitor Health Routines, such as “Percent of Parents Reading Daily with their Child.”

Figure 1a. The Dashboard monitors resident experiences and trends in the community overall and in particular Magnolia Initiative partner settings. This chart shows a sample of a Care Process measure, which assesses the extent to which organizations are carrying out the work that the Initiative partners believe will improve outcomes. Goals (dotted green line) are set at ambitious levels that are considered achievable. Data are self-report measures from monthly client surveys collected by organizations. The Dashboard also includes charts that monitor Health Routines, such as “Percent of Parents Reading Daily with their Child.”

Breadth and depth of data are no less important. Agencies typically collect and report on only their actions or outputs, for instance, counts of clients served. Yet, this level of data does not reflect what families or individual staff experience when receiving or providing services, and it sheds little light on why or how those services result in change (or do not result in change, as the case may be). Data that capture the experience of program users or participants can give providers a better sense of the true results of their efforts. This means getting more granular than looking at counts of services delivered at the front end and then tracking high level “outcomes” such as rates of obesity or high school graduation. It means instead that outcome data should be coupled with information about individual experience as well as family and community context, all of which affect how well the system can function to achieve better results.

The Initiative uses a data dashboard to synthesize and make these multiple levels of data accessible for partners and others. The dashboard displays a range of items from day-to-day actions (e.g., rates of daily reading to children) to care processes (e.g., how frequently service providers ask clients about family stressors) to the family conditions that ultimately drive outcomes (e.g., family stability, food security, access to needed care and supports).The dashboard serves as the data visualization for our theory of change and functions as a tool to coordinate efforts and foster shared learning and joint accountability for results. It displays the various measures chosen by the partners to reflect their agreed upon actions and the intended short-term impacts (see Figures 1a and 1b). Short-term impacts serve as indicators for the longer-term changes, and ultimately these long-term changes are positive indicators that the efforts are leading to community-wide change.

The Magnolia dashboard differs from many others by having real-time and regular measurement, much the same way that the dashboard of a vehicle displays both current speed, indicating how the driver is doing at the moment, as well as miles accrued, indicating cumulative progress. We find the dashboard holds promise as a tool to align leaders and organizations around collective action and long-term results.

Figure 1b. The Dashboard also displays results of a biannual survey of community residents that assesses protective factors: social connections, concrete supports in times of need, and individual resilience. The chart shows composite measures in blue, representing the percent of respondents affirming all measures in a given category. The light green represents the percent of respondents affirming the noted sample measure from each category.

Figure 1b. The Dashboard also displays results of a biannual survey of community residents that assesses protective factors: social connections, concrete supports in times of need, and individual resilience. The chart shows composite measures in blue, representing the percent of respondents affirming all measures in a given category. The light green represents the percent of respondents affirming the noted sample measure from each category.

Figure 1b. The Dashboard also displays results of a biannual survey of community residents that assesses protective factors: social connections, concrete supports in times of need, and individual resilience. The chart shows composite measures in blue, representing the percent of respondents affirming all measures in a given category. The light green represents the percent of respondents affirming the noted sample measure from each category.

Data Can Promote Learning and Change

In determining the scope of information needed to support community change, the Magnolia Initiative partners began by adopting a set of measures that tells the story as they imagined it would unfold based on the network’s theory of change. That theory holds that the outcomes for a population of children and families depend on the day-to-day actions of individuals and organizations supporting families and other neighborhood residents. Progressive changes in these actions contribute to shifts in family and neighborhood conditions as well as changes in individual health and parenting behaviors. Taken together, all of these small shifts build toward the longer-term improved outcomes for children. As such, the partners have constructed a measurement system that aims to concurrently capture information about each of these elements. The measures are clustered in the following categories, indicating which aspect of the system they shed light on:

  • Actions of individuals or organizations to support positive behavior change. Examples of these types of measures include whether providers or staff asked residents about their well-being, concerns about their family, and whether they were guided to local resources to address their concerns. Another measure asks whether service providers and staff were able to respond to needs expressed by residents or clients using their own resources or by linking to other local organizations.
  • Conditions of families and neighborhoods that enable people to sustain health-promoting behaviors. Examples include residents reporting that they have social connections with other residents or social support for personal needs or problems, sufficient food, feelings of safety in the neighborhood, safe places for children to play, enough money to prevent frequent moves, and positive role models for parenting.
  • Individual behaviors that contribute to positive health and economic outcomes. This includes asking residents about behaviors that promote health, such as physical exercise and healthy eating, in addition to daily habits such reading books to young children and other parenting routines that are associated with children’s learning and development. Residents are also asked about the extent to which they have regular contact with family and friends and know their neighbors.
  • Outcomes or results the system is producing. Examples of these measures include the social and emotional well-being of children as they start school, the percentage of local children who are proficient in reading in third grade, the percentage of children in middle school reporting that they have positive relationships with peers and adults, the rates of children graduating from high school, and rates of child abuse or neglect.

The Initiative partners have also included other measures such as “reach” to understand the extent to which the target population is linked into the network or system, and therefore experiencing the improvements. For example, if several health clinics improve their services, but together serve only 20 percent of the local population, that helps explain why changes might only be seen for a subset of the community.

Data that only helps to identify a problem is not adequate. The data must also help fill in the details of the possible pathways for addressing the problem. Without the latter, organizations find it difficult to use the data when designing system-level improvements. For example, test scores can show that students are not reading at grade level. Although a group might have additional information about the extent to which the children are receiving tutoring and test-day breakfasts at school, other influences—such as study habits, family stability, and parent interest in their child’s academic success—can be even more important to both the near-term (test results) and long-term (high school graduation) outcomes. Without data on those facets of student life, it’s impossible to gauge opportunities for improvement. If, in drawing boundaries around our problem set, we fail to consider the potential causes of a situation and lack information that allows us to test our assumptions, we can miss critical intervention opportunities.

Data Needs Vary across the System

We’ve learned through our experiences that the different data needs of different types of actors in a system must all be taken into account in designing a measurement approach. Leaders in a system require a set of measures that indicate whether and how the day-to-day operations are achieving a larger impact – e.g., headline statistics such as third grade reading scores or rates of low birth weight births. But doctors, social workers, classroom teachers, and parents need other kinds of data to guide actions within their sphere of influence. They need the data that help them achieve the specific day-to-day actions for which they are responsible, and that contribute to these long-term results along with the actions of many others. By providing the various stakeholders information that is relevant to their particular roles, everyone receives the information they need to take actions within their realm of responsibility or influence. Putting the measures together in a dashboard allows everyone to see if everyone’s collective actions are resulting in meaningful change. Consider the different data needs for the following stakeholders:

  • Policymakers, system leaders, and public agency leaders have the responsibility for systemwide outcome measures and results. This refers to numeric targets (e.g., 90 percent of children enter kindergarten ready to learn) for what success would look like. Beyond policymakers and system leaders, community members and stakeholders can also use these outcome measures as motivation for their own actions. This may involve advocating for improved neighborhood and larger community conditions.
  • Organization and program leaders focus on operations and strategy for their particular organization, and need information about whether actions and practices among staff are contributing to the intended goals and aims of the organization. For example, a strategy might call for empowering clients to take charge of their health. Survey questions that ask clients if service providers paid close attention to what they were saying and let them tell their story without interruption help identify whether service providers are providing care in a manner that helps clients have a voice in the process. If not, it is the responsibility of managers or program directors to design training or coaching strategies for staff so that the system is delivering on actions that are consistent with an overall strategy.
  • Frontline providers (such as doctors, social workers, teachers, and intake workers) need access to regular and timely data that offer feedback about the care they provide. They often need this information daily, weekly, monthly, or soon after the service date so they can easily see if their actions are leading to an improvement. The data that are gathered monthly and quarterly inform day-to-day actions that support improvements in practice. Frontline staff gather and receive timely and consistent feedback through surveys that ask clients about their experiences with the care received. For example, one of the goals of Magnolia is to routinely identify and respond to family stressors. Clients are asked during all encounters with providers and staff if anyone asked them about any changes or stressors in their home, and whether they received information about other programs that could be helpful. Reports are then shared with providers and staff to show that, when repeated, these measures can detect whether intentional changes in practice are producing the desired result. If not, the data can point to areas ripe for adjustments that might produce a better result, such as staff using a checklist as a reminder or prompt to take certain actions. Although this type of report and data display has rarely been available to frontline providers or managers, we have found that it is a powerful motivator to improve practice. Although the social sector has just begun to use these types of methods, there are numerous examples in the business and health sectors of employing real-time data to improve practice. Ironically, frontline providers often spend considerable time gathering and entering data, and yet they often do not receive it back in a meaningful time frame, or at all.

How Community Members Can Use Data

Community members use data differently depending on their focus. Measures may be relevant to them in their capacities as neighborhood residents; as participants in services; as clients working in partnership with organizations; as parents striving to create safe, nurturing and development-promoting home environments for their child; or all of these.

In their capacities as neighborhood residents, community members have found it helpful to know the actual counts of children in their neighborhood who are “on track” in health, development, and learning. Knowing this generates engagement and concern among residents. However, to identify which steps to take to change conditions, residents also need information about the factors that shape children’s developmental progress. The Magnolia Community Initiative uses a holistic measure of child health and well-being—the Early Development Instrument (EDI). The EDI is a measure of children’s development at age 5 based on a checklist completed by kindergarten teachers. The EDI consists of more than 100 questions measuring the following five developmental areas: (1) physical health and well-being, (2) social competence, (3) emotional maturity, (4) language and cognitive skills, (5) and communication skills and general knowledge. These areas correlate closely with nationally accepted measures of school readiness.

EDI results and other measures of family well-being in neighborhoods are geocoded and mapped to the local neighborhoods using boundaries defined by local residents. Examples of these measures fall in the realm of social cohesion (e.g., are people willing to help their neighbors?) and informal social control (e.g., would neighbors do something if children are hanging out on the street? Do the parents on a neighborhood block know one other?).

Mapping and publicizing these measures help galvanize direct actions that can alter family and neighborhood life because the data can help neighborhood residents organize around broadly shared concerns and perceptions. The information also allows residents to see how their neighborhood compares with others in terms of the well-being of children and adults. Some have used the information to advocate for changes or new resources in their neighborhoods. For instance, community members have used information to inspire others to participate in Neighborhood Watch. Others have been inspired by data to change their personal behavior, such as exercising more or reading together with their child. It is important to support community members with the right information tailored to their purpose. The type of information that motivates one to advocate for resources or safety should not be assumed to be the same type that is needed to help parents change their personal behaviors at home. Motivation to advocate can come from neighborhood-level measures of well-being, whereas motivation to change one’s own personal actions requires measures of individuals’ daily activities and their impact.

In their capacities as parents striving to create positive home environments for their children, parents have benefited directly from information on whether they are meeting their own specific goals. For example, child care providers and others can track how often parents are able to read books with their young child, and then can help parents with goal-setting and tracking progress. This is a similar process that organizations follow in trying to effect a change. While the measurement process and the display of data can be tailored to the specific user, the concept of individuals having regular information about their goals and progress toward those goals is as relevant to a parent as it is to an organization.

Achieving Scale and Sustainability

This kind of a data system can only work when it is scalable and sustainable, such that all network partners who need to be involved to achieve a system-level, community-wide change can actually adopt measurement as part of their routine practice. This means ensuring that the data system is affordable, contains measures that are relevant across organizations and sectors, and is not burdensome in the time and resources required for regular data collection. In addition, the number of measures being tracked must be limited so as not to overwhelm users.

There are well-recognized features that can serve as guiding principles for selecting useful measures to drive change for programs and systems as well as for community systems that are striving for a population impact. Valid measures represent the intended concept. Reliable measures dependably gauge the intended concept over time. Measures should be sensitive to change so it is possible to know whether actions taken to improve a process or experience are producing better results.

In deciding which measures to collect, we have also found that it is important to consider what measures will motivate people to focus on collective action. Individuals and organizations need to see how their contributions contribute to the shared goal. This helps move from an individual or client perspective to a collective responsibility for population outcomes.

It is also essential to consider scale when the intention is to have an impact on a geographic population of children and families. Working at scale means that the data collection needs to be feasible for all involved organizations, under all conditions. Some data collection systems that track information about children and families are proprietary and costly to organizations. Data systems that are costly to maintain, or difficult for organizations to use, are unlikely to be adopted by the number of organizations required to achieve a population change. Also, data systems adopted for a specific, time-limited project or a specific sector that are not designed with other goals and sectors in mind are unlikely to meet the needs of long-term, community-wide change processes.

The key is to have measurement become a routine part of practice and regularly reviewed to inform decision making, assess progress, and support improvement over time. To that end, the measurement system must be designed for the long term. It is important to sustain the measurement long enough to see the learning, testing, and implementation take effect.

Measurement that takes too much time to collect and that is not built into the regular process of caring for clients or one’s daily routine, or is simply overwhelming for leaders and practitioners to track, is unlikely to be sustained as a permanent part of a community system. At the end of the day, all measures come at a real cost, so it is essential to choose measures carefully. Although almost counterintuitive in today’s era of “big data,” selecting a modest, balanced number of measures (10–15) is much more effective than larger sets.[2]

Core Principles for Using Data in Complex Systems

By introducing a disciplined approach to learning, organizations can move quickly from an idea for change to testing and implementing that change. In this way, the network “acts” rather than “plans” its way into a new system.[3] A systemic approach to learning equips partners to adopt new practice and policy as it emerges. It enables organizations to translate cutting-edge ideas into care processes that work across many people and organizations, even those in the most challenged settings. A critical part of this process involves attending to human reaction to change. Introducing measurement without attending to personal aspects of what helps people change their actions and behavior is enough to limit the impact of even the most robust theories and approaches. It’s an old saying that while everyone wants progress, nobody likes change! As such, we have found that more meaningful and lasting change can occur when path-breaking ideas are introduced in conjunction with coaching on ways to take action.

The purpose of measurement is understanding and reflection, plus change. To that end:

  • Select a set of measures that reflect your theory of change from action to results, being mindful not to overwhelm those you hope will participate by adopting too many measures.
  • Design a measurement system for scale so that all the community members, community-based organizations, and decision makers who are important for the outcomes will receive the information they need to take the appropriate actions at their level. This also means attending to cost considerations to develop, maintain, and participate in a data collection system, as well as considering open versus proprietary information systems.
  • Design for sustainability so that the data support an enduring change process. Designing measurement support for a two-to three-year process may inform some change, but it is unlikely to deliver community-wide change or provide families with a supportive environment for a long enough time to improve their conditions, actions, and well-being.
  • Consider all the different roles that community members play when providing or helping gather and use information. They are a critical voice in showing how a system is performing and in creating the demand for change. Recognize that information for one purpose may not be helpful for other purposes, with the understanding that not all community members will play all possible roles.
  • Avoid giving people (actors, stakeholders) measures without a change process that helps them to take actions in their sphere of influence. It is also essential to shift to providing timely, monthly progress on process of care measures overall and by service sector to provide diverse programs and providers both shared accountability and a common change process. Understanding variation within an organization, or across a system, is a cornerstone of effective improvement.
  • Offer coaching and other support to make a change. Remember that having information alone is insufficient to drive change in professional practice, in resident actions, or in personal behaviors.
  • Be ready to change both measures and strategies if they appear not to be as informative, effective, or change-inducing as predicted.

Opportunities Going Forward

A key design imperative is to collect data in a way that meets the needs of the user, which in a change effort is the entity or actor responsible for the targeted change. We know from our experience in supporting population change efforts, such as the Magnolia Community Initiative, that program directors, evaluators, and leaders of community change efforts are better able to support change when they use methods of data collection, display, and analysis that are designed for dynamic systems. Fortunately, the cost and ability to gather data are shifting enormously, creating opportunities to use data and measurement in new ways.

We have seen a growing appreciation that small data samples repeated more often are not only more feasible than conventional evaluation methods but also more powerful for real-time learning and knowledge. As the value of process improvement becomes more evident to stakeholders, comfort with using time series and real-time data to support frontline providers will increase. A greater appreciation of the value of these methods can provide an important opportunity to equip all actors with data they need for action and for gauging change over time.

There is also great value in collecting and sharing resident and client voice in a productive way, for reflection and action at all levels of the system. For residents and practitioners in community serving organizations, measures of neighborhood conditions can be much more motivating and useful than regional, city, or countywide measures, so there is a need to help residents and others secure these data. For neighborhood change, it is imperative to help residents identify and use measures and data collection processes that align with their needs and that are affordable and sustainable for the period of time they will need them.

Lastly, while it is often stated that “what gets measured gets done,” it is also common knowledge that “you can’t fatten a cow by weighing it.” Data alone are insufficient for driving significant change, particularly in complex community systems. Rather, more attention needs to be placed on creating learning environments. This means that in addition to support for data collection, there must also be support for efforts to make meaning of the data, make a prediction, and learn our way forward. Many fields offer examples of collaborative and iterative learning methods that can accelerate innovation and improvement.[4] What these efforts have in common is a successful transition from using data for generating information to using data for reflection and change.

[1]   S. Glouberman and B. Zimmerman, “Complicated and Complex Systems: What would successful reform of Medicare look like?” Discussion paper No. 8. (Commission on the Future of Health Care in Canada, 2002).

[2]   G. Langley et al., The Improvement Guide: A Practical Approach to Enhancing Organizational Performance (San Francisco, CA: Jossey-Bass, 2009).

[3]   A.S. Bryk, L.M. Gomez, and A. Grunow, “Getting Ideas into Action: Building Networked Improvement Communities in Education.” In Frontiers in Sociology of Education, edited by M. Hallinan (New York: Springer, 2011).

[4]   Ibid. Also see C.M. Lannon and L.E Peterson. “Pediatric Collaborative Networks for Quality Improvement and Research,” Academic Pediatrics 13(6 Suppl)(2013):S69-74; A. Billett et al. “Exemplar Pediatric Collaborative Improvement Networks: Achieving Results.” Pediatrics 131(Suppl 4)(2013):S196-203.