Can local organizations make their neighborhoods stronger and healthier? If they can, how do they provide evidence that they are making a difference?
These are important questions, especially as millions of data points on neighborhoods become accessible via mobile apps, Web portals, and powerful databases. But these questions are not new. Former Federal Reserve chairman Ben Bernanke made the case for data-informed analysis of community work at the 2003 Community Development Policy Summit in Cleveland, suggesting that groups could raise funds and other types of support more effectively if they would “capture intangible social benefits, such as those that accrue to a neighborhood as residents become engaged in community planning activities, improve their financial literacy, and increase their access to employment opportunities through job training.”
Using data to demonstrate these types of effects has become a higher priority in recent years, as both foundations and government seek validation for the work they are supporting. But it’s a tricky proposition, because first you have to show that something good has happened and then link those good results to a specific community improvement strategy, as opposed to a strategy or force from outside a specific organization’s control.
Just as important is the underlying framework for mounting the community improvements: the “logic model” or “theory of change.” Jane Jacobs and Robert Moses went to war over these frameworks way back in the 1960s. Jacobs, Greenwich Village resident and author of The Death and Life of Great American Cities, argued that the complex networks of relationships in New York neighborhoods must be preserved and nurtured, whereas Moses, New York City’s construction czar, pushed for highways and urban renewal to bring economic benefits. In his classic 1998 book, Seeing Like a State, James C. Scott outlines numerous calamities caused by centralized planning that failed to incorporate local knowledge and marshal local capabilities. In addition to Soviet and Chinese failures, and the Brazilian folly of building its new capital on the country’s distant central plateau, he recounted the data-driven efforts by 18th-century German forest scientists to create highly productive woodlots with a single species of trees, optimally spaced and free of competition from other trees. This experiment, like later efforts in the United States to build densely populated, isolated housing developments for poor people, sought to bring order to a complex reality, but led to failure.
After several decades of urban renewal in the United States, the weaknesses of centralized, top-down approaches became apparent, prompting new experiments to support “locally driven development.” This framework departed from both top-down central planning and bottom-up community organizing. It moved nonprofit community groups into ownership and development of affordable housing and shopping centers, which required a business mindset and appealed to early supporters including the Ford Foundation and Senator Robert F. Kennedy. Nonprofit builders notched decades of successes, producing impressive outputs of housing units and commercial space, but eventually this too proved inadequate in the face of larger urban challenges. From that realization emerged the “comprehensive” framework, which folded education, health, safety, recreation, and other quality-of-life issues into the portfolios of community development corporations.
The comprehensive model has driven the work of Local Initiatives Support Corporation (LISC) Chicago and many other organizations for the past decade or longer, creating a rich dynamic that weaves central intermediaries such as LISC with community-based nonprofit organizations, social service agencies, local governments, foundations, corporate supporters, and subject-area experts on various issues.
This is complex and exciting work that appears to produce good results. But we still must ask, does comprehensive community development improve neighborhoods? And can we prove it?
Data and Technology
Rapid technology advancement and ubiquitous data create opportunities to begin answering those questions. But doing so will require strategies to address the same type of top-down, bottom-up tensions of earlier debates, as well as a clear understanding of the frameworks and theories to be tested. Perhaps most important, answering these questions demands a “data capacity” that is only beginning to be developed at the community level.
LISC Chicago has been nurturing this new capacity throughout its neighborhood work since the late 1990s, when it began emphasizing knowledge creation and journalism-style reporting around its comprehensive neighborhood initiatives. The early work was largely qualitative—more narrative than data portrait—but recently the data streams have been providing useful information about the reach and impact of LISC’s work. Examples include the following:
- Data after the fact: Near a rough corner on Chicago’s West Side, Breakthrough Urban Ministries has, since 2009, hosted Friday-night basketball tournaments that attract scores of youth and families. Whether it made a difference in crime was not clear until maps and charts—winnowed from millions of lines of police statistics on the City of Chicago Data Portal—showed that nearby crime had fallen for three consecutive years.
- Data for program tracking: Six weeks after the launch of the Affordable Care Act, LISC’s partners in 24 neighborhoods had used the Web-based tool Wufoo to log contact with 4,314 people provided with health insurance information. (This number exceeded 14,086 by January 2014.) The shared, private database shows contact information, ages, language preferences, and locations, which helped insurance outreach workers track enrollment, flag individuals for follow-up, and develop program improvements. The ages alone offered important information, as the system needs enrollment of young adults to help balance the risk of older and less-healthy enrollees.
- Data for understanding: Surveys created by 10 community-based partners in the Little Village neighborhood provide detailed and, sometimes, disturbing portraits of at-risk youth, covering family life, academic attitudes, and stress related to violence at home or on the streets. In this low-income neighborhood composed primarily of Mexican immigrants, the information helps agencies better understand the youth they serve and provides a framework for more collaboration and coordination in youth programming.
What’s remarkable about these data-informed snapshots is that they were generated by LISC-supported community groups working on the ground, not professional researchers. Until recently, regional think tanks and “data shops” were the dominant purveyors of data because of their ability to access census tapes and other institutional data, which they filter and demystify for clients in city government, neighborhood organizations, or foundations. Today, trillions of data points can be accessed by just about anyone. New tools and technologies are invented nearly every day to collect, sort, and display the information. Responding to these opportunities, many nonprofit organizations in Chicago have caught the “data bug” and begun to recognize the importance of using data to improve programming and demonstrate impact.
But it’s a steep learning curve to make sense of it all—and to do it right. At the entry level, an organization must develop basic skills to interpret the data within a framework or theory of change. Such skills transcend simply manipulating Excel files. Rather, practitioners must learn at least one database, develop an ability to formulate query language, and know how to extract useful reports.
Next, the ever-evolving technical tools must be mastered. These tools include visualizations, “dashboards,” and mobile apps for inputting or sharing data, all of which require technical and software skills. As organizations delve into the data, they discover the intricacies of survey methodology (particularly when tracking over time), the strict protocols for human-subject research and waiver requirements, and, in the health field or when dealing with schoolchildren, legal and privacy requirements that may hinder data collection and use.
This isn’t something that can be done on the side or without commitment from an organization’s senior leadership and the dedication of its staff. Although consultants may help, the work cannot be entirely outsourced. At least one person in the organization must have an innate curiosity about the data, and that person must be willing to experiment with data tools and demonstrate to colleagues the value of developing data-informed narratives.
Building Local Knowledge
For some nonprofit community organizations, a key driver for taking on these daunting challenges is economic survival. The prolonged recession caused many philanthropies to question whether their investments had any lasting impact—particularly if larger economic forces like the recent foreclosure crisis could undercut decades of incremental improvements at the community level. The recession also reduced endowments at foundations, cut government tax revenue, and trimmed corporate profits, leading to significant cuts to funding for nonprofit organizations. Faced with tougher decisions about which groups to support, funders demanded more evidence and data from organizations on not only “outputs,” such as people served or dollars spent, but “outcomes,” which require sophisticated logic models and the data to show what is happening.
Everyone, of course, wants to know if their work is having an impact. But the reality has been that most nonprofits simply have not had the expertise or capacity to capture data, sift the information, analyze and discuss the results, improve programming based on the numbers, and then prove effectiveness. More often, as foundations and government have demanded more metrics for the programs they fund, the harried nonprofit organizations have dutifully provided information, sometimes devoting entire days to data input through multiple spreadsheets or databases. Unfortunately, these organizations rarely have had the time or inclination to find useful lessons in the data. Nor have they used data meaningfully at the front end of program development to inform strategy or monitor implementation.
LISC Chicago was one of these nonprofit organizations, more interested in achieving results than proving a causal relationship. Even so, over many years of trial and error, LISC Chicago has gradually become more involved with data and its various uses. The interest was driven by more than a decade of learning from LISC’s demonstration of comprehensive community development, the New Communities Program (NCP). The program used a relatively consistent methodology that has always included a respect for information flows, a bottom-up and top-down process, and strong support for knowledge building by NCP’s primary funder, the John D. and Catherine T. MacArthur Foundation.
The New Communities Program began as a pilot in three neighborhoods in 1998 with nearly a year-long quality-of-life planning process that collected information from residents by using markers and newsprint sheets. The planning participants discussed and sorted these data into issue areas that could be addressed by interlocking strategies. A lead agency coordinated the neighborhood efforts, and in each plan, a chart showed the multiple neighborhood partners who had agreed to lead projects in their areas of expertise. As the plans were implemented, coordinated sets of projects reached residents and produced concrete, visible improvements in the neighborhoods, like new employment centers, a community newspaper, youth-painted murals, and new retail development.
This was considered a sufficiently major accomplishment that, in 2003, the MacArthur Foundation supported a $17 million, five-year demonstration of the NCP method in 16 neighborhoods. Everything began with community engagement and planning, including some nicely bound data books that provided dense demographic and education tables, much of it from the U.S. Census, to help inform development of strategies. LISC contracted with urban planners to help guide the process, and hired former journalists—known as scribes—to document the discussions and create a coherent narrative about the neighborhood, its assets, and its challenges. The data books were ultimately not used extensively, but the plans incorporated a significant amount of local knowledge—what might today be called “little data.” Those plans ultimately leveraged more than $500 million in new investments in the NCP neighborhoods and led to documented program-level outcomes in the areas of education, Internet use, and income- and credit-building.
When this work started, LISC and its partners were scouting the foothills of data. Navigation was mostly by instinct, as was program design and execution. LISC had partnered with a diverse set of community organizations from a range of low- and moderate-income neighborhoods. These groups could build partnerships with other neighborhood organizations and they had a stable commitment of operating and program funding from the MacArthur Foundation via LISC, so they were able to mount hundreds of small and larger projects in the first five years.
If someone had asked for proof that the MacArthur Foundation’s initial investment was making a difference, LISC would have assembled a library of journalistic stories, thousands of professional photographs (another form of data), and a few charts showing where the money was spent. Local leaders routinely provided site visits at thriving new employment centers or rebuilt public spaces, and they used Web sites, fliers, maps, and reports to communicate that things were working and “producing impact.”
But hard, organized data? The most important data collected early on were self-reported estimates of leverage, loosely defined as new investments in the neighborhood that were connected to specific strategies or projects in the quality-of-life plans. Some of these numbers were impressive, suggesting that the plans and their networks of partners had improved the “capital absorption capacity” of the neighborhoods. But the data were unable to demonstrate causality—that the program caused a reduction in local crime or an improvement in graduation rates, for example—and were rarely consistent enough to support a theory of change.
A more subjective outcome, documented by the evaluation firm MDRC working for the MacArthur Foundation, was the creation by NCP of a platform of collaborative networks, financial resources, and technical assistance that produced significant positive activity in the NCP neighborhoods. These networks had actually implemented the quality-of-life plans through discrete, concrete projects and programs, something that many previous comprehensive initiatives had been unable to do at scale. However, sufficient data did not exist to accurately track outcomes or correlate the projects with changes in traditional data sets being tracked by MDRC, such as mortgage originations or small-business loans.
These trends began to change during the second five-year commitment by the MacArthur Foundation. As the foundation’s total investment grew to nearly $50 million over the ten-year period, LISC raised an additional $50 million from other funders. Some of those funds supported three multi-neighborhood programs that successfully integrated data use into the everyday rhythm. The number-crunching was tedious at first and required significant investment in database development, training, technical assistance, and supervision. Alongside the development of data expertise, LISC continued to emphasize storytelling and communications, deploying its own scribes to create meaning from the data. LISC also provided training and consultation to help neighborhood organizations develop these skills internally. Over time, as data were used to tell compelling stories, skeptics began turning into believers. The following are a few examples:
- Income- and credit-building: At 13 LISC-supported Centers for Working Families, participants are offered three core services: job placement and career development, one-on-one financial counseling, and enhanced access to income supports. Close tracking and incisive analysis found that those participants who used two or more services are eight times more likely to increase net income than those who receive only one service. This result, confirmed for multiple years, has led to stronger integration of multiple services. Most recently, the centers added a fourth component, digital skills training, which is showing another layer of evidence in the form of improved job placement rates.
- School attendance: A multiyear commitment by the Atlantic Philanthropies allowed LISC partners in five neighborhoods to extend school days, add in-school health centers, and provide family supports at local middle schools. Close tracking of health data, including immunizations, showed that many students lacked the immunizations they needed to stay enrolled in school. Organizers identified these children and referred them to the health center for immunizations and health exams. Attendance rates, essential for academic gains, rose soon after, proving the positive impact. The neighborhood partner in Auburn Gresham was so impressed by the results that it analyzed immunization data at nearby elementary schools and rented buses to bring students to the health center. Again, attendance showed an upward trend. (This work required signed releases from parents of all students, which provided an important lesson on how much time and effort it takes to honor privacy and health information laws.)
- Digital skills: A federal stimulus grant to the City of Chicago funded intensive outreach and training that was coordinated by LISC Chicago in five “Smart Communities,” where neighborhood “tech organizers” promoted and taught classes in basic computing, Internet use, and office skills. Baseline survey data found that people avoided the Internet because of lack of interest, high cost, and difficulty of use, so the program was designed to provide free Internet access at neighborhood centers and hands-on training in basic tools such as e-mail, social media, and Microsoft Office. Early adopters became enthusiastic users and promoted the program to family and friends, leading to waiting lists and expanded offerings that produced 7,000 course completions and 17,000 visits per month to community Web portals. Formal evaluation in 2012 showed a real impact: a 13-percentage-point gain in Internet use compared with similar neighborhoods nearby. The City of Chicago is now promoting expansion of the model citywide.
In all of these cases, collecting and analyzing data served two distinct functions. First, they provided real-time information about program execution (the number of people served, in what capacities they were served, what types of services they received), which allowed program managers and their supervisors to identify strengths and weaknesses, to enforce accountability among local partners, and to implement program adjustments. Second, they created more meaningful data that allowed community-level program managers and professional evaluators to measure impacts. This documentation gives LISC Chicago confidence that its comprehensive, community-based efforts are making a quantifiable difference.
Extending the Method
The NCP method emphasizes the importance of local leaders having a voice and agency in decisions that affect their community. In today’s world, this requires supporting local leaders’ ability to interact with data systems and to apply the resulting information to the daily work of community development.
Capacity-building for data, from our point of view, doesn’t start with data. It starts with the fundamentals of community engagement and planning, which of course are grounded in information about the neighborhoods. It starts with a methodology that brings in outside partners with data and tech expertise to add value to the community partners, who have their own local knowledge and program implementation expertise. It requires building sufficient trust so that local actors can safely unpack and question their own strongly held assumptions and theories about what works. It means having an entrepreneurial approach that allows innovative ideas to be tested and evaluated, to see if they work, and to be respectful if the answer is, “No, they don’t.” And it recognizes that such capacity requires sustainable investment in an additional layer of information infrastructure that is beyond direct program costs. Unfortunately, building data capacity does not change the fundamental challenge of sustaining programs that rely on diverse streams of public and private funding.
After more than 10 years of extraordinary investment in and partnership with the New Communities Program, and only limited documentation of effective implementation in the official MDRC evaluation, the MacArthur Foundation asked LISC to further test its community development approach; to ramp multiple neighborhoods to a higher level of data use and evidence-based practices; to use that data to inform program design; and to track progress toward community-level change.
LISC responded by developing a knowledge-driven approach, called Testing the Model (TTM), which builds upon the NCP method and embeds data collection into focused strategies that neighborhood partners choose and tailor. Each plan includes a “theory of development,” a series of related interventions, and datasets that help track activities and progress.
Knowing that it needed to build its own skills and those of its partners, LISC expanded its relationships with data-shop partners such as Chapin Hall at the University of Chicago and DePaul University’s Institute for Housing Studies, which would offer guidance on data approaches, evidence-based practices, and research methodologies.
LISC worked intensively with a small cohort of its neighborhood lead agencies to develop data-informed community plans and the data capacity to effectively implement the plans and track the results. This involved significant amounts of time by program officers and scribes, who participated with the neighborhood groups and their data partners in a series of meetings that lasted for a year or longer. In each community, the budget covered support for a full-time program lead, a part-time data-entry specialist so that the data tasks would not distract from the core work, and seed funds to launch new strategies aligned with the plan.
The resulting collaborations provided learning experiences for all involved. The data and academic consultants had rarely worked so closely with community-level partners in the development and early implementation of programs, and they found these close relationships were more fruitful than the usual after-the-fact, arms-length observations. In the neighborhood, the initial reaction to the data partners was typically wary because community groups neither spoke the language nor had a background in data methods. Many early meetings included awkward moments and furrowed brows, including defensiveness among the neighborhood participants about being “pinned down” and made accountable for showing progress in particular ways. In time, however, as appropriate and meaningful data points were selected for tracking and new projects from the plan launched, neighborhood groups and their data partners became excited about what was being built.
Spreading Data Skills
While the TTM approach was being developed, LISC was building on its other data experiences with Centers for Working Families, the Atlantic Philanthropies middle-schools effort, and the Smart Communities demonstration. Sensing a desire for data expertise in its neighborhood network, LISC instituted a monthly series of informal gatherings called Data Fridays during which self-described data geeks explain their work to diverse and lively groups of 20 or more neighborhood development people. LISC also invested staff and consultant time to become familiar with powerful new tools such as the City of Chicago’s data portal and the various apps being created by the city’s open data community.
These continued investigations reinforced LISC’s understanding that collecting data or digging into data sets was only the first step. Routine and useful application of data would require not only analysis but the artistic skills required to develop infographics and other visualizations, in addition to higher levels of technical knowledge to influence or design Web and mobile tools that facilitate data collection, retrieval, and presentation. A grant from Boeing helped LISC delve deeper in these areas, most recently through a contractual engagement with a civic tech firm called DataMade, which specializes in creating vivid Web-based charts, maps, and tools related to crime, housing, and other civic data. This agreement is helping not only LISC but other neighborhood groups to become more involved with use of data.
After wading into the arcane world of civic hack-a-thons and “open government”—where discussions are laden with technical language about application programming interfaces (APIs), back-end databases, and URL query strings—LISC and neighborhood partners won support from the Knight Foundation for “Open Gov for the Rest of Us,” which is helping a few of LISC’s neighborhoods develop the technical language and data skills necessary to build bridges to Chicago’s thriving tech and civic hacking scenes. The project provides funding and technical support to neighborhood partners who engage residents in trainings and discussions about local issues and how data might be used to address them.
Building Data Culture
Like many in the community development field, LISC and its partners are beginners in the use of data to inform and improve its programs. Over the years, we have learned a great deal about what it takes to build a data culture and how it can spread within and among organizations. Six lessons stand out:
- Community groups need to develop “data and information capacity.” A June 2011 assessment by the Metro Chicago Information Center found that most of LISC’s neighborhood partners (and LISC Chicago itself) had very limited capacity to collect and analyze data. Computer systems and databases were often inadequate, and many partners had limited or no in-house data expertise. LISC and many partners recognized the value of embracing data, however, and made commitments to build capacity through formal and informal methods. As LISC has directed financial, training, and technical resources into data capacity-building, many types of neighborhood partners have made significant progress on the data continuum. (This progression mirrors work performed five to seven years earlier when LISC and partners improved their digital communications skills including use of Web sites, social media, and video.)
- The right kinds of technical assistance are well received. Most organizations accepted the initial assessment without becoming defensive or feeling that data expectations were pushed on them by an outside force. Instead, they recognized the opportunity and many immediately made small changes such as improved computer networks or more attention to existing data collection. When LISC offered technical assistance from a variety of data support organizations, community groups responded favorably and shared their own lessons with peers. One neighborhood has even instituted its own “data geek squad,” composed of senior staff members with strong data skills, a dedicated part-time data-entry specialist, and outside data consultants, to support other partner agencies.
- Peer-to-peer learning works. One of the least-threatening ways to spread data skills is through informal peer sharing, which can range from LISC’s Data Friday gatherings to one-on-one encounters that demonstrate how a simple chart or visualization can bring data alive. LISC has found that community developers are hungry for this new knowledge and are excited about using it. Furthermore, LISC has learned that the best way to spread data culture is to create venues and programs that expose more people, at various levels, to data that are relevant to their work.
- Good enough is good enough, to start. Although early attempts to use data will likely be awkward and perhaps inconclusive, the only way to gain expertise is to experiment with the data. Many software developers use an “agile” approach that begins with a simplified working model that is then used, refined, used again, expanded with new features, and finally built into a fully tested product. The same approach has worked for development of the neighborhood data plans and the databases that support them. Unfortunately, philanthropic funding does not tend to follow this agile approach. Patient capital is needed if nonprofit organizations are to commit to learning and performing the data work.
- Data culture (or the wrong partner) cannot be forced on a group or individual. Although some groups and individuals are responsive to data, others may not be. LISC and its partners have experienced multiple instances of incompatibility or poor timing in which individuals have resisted or rejected data roles, and several organizations have had unsuccessful matchups with data partners. LISC used the initial capacity survey to gauge the readiness of partners, choosing the most “data-curious” for the first round of investments. The important test came when the work started and particular staff and partners had to collect and find meaning in the data. When that didn’t work, in most cases, another attempt with different people and different partners—properly selected and supported—led to successful uptake of the data culture.
- The work still has to get done. To varying degrees, nonprofit partners with whom LISC has worked to integrate data into local programs have shifted their attitudes on the “burden” of data collection and analysis after having seen how it helps them discuss and learn from their programs. But for some groups it slowed the work to nearly a standstill as they struggled with partners over evidence-based models and data-sharing protocols. For others, a highly structured data-and-outcomes-driven plan did not align with the loosely structured, but highly productive local program design and staffing. The power of data to tell stories and prove impact only matters if the program gets implemented, residents benefit, and the work gets done. Along the data-capacity-building continuum, LISC recognizes “data-inspired” and “data-informed” as reasonable ground for community development practitioners to stand on.
How Data Are Changing LISC Chicago
LISC, with its partners, is learning how to use data to improve program design and implementation, to support the comprehensive development of neighborhoods, and to improve the quality of life of residents who live there. Although it is still not possible to prove that every program or approach is creating a particular measurable impact, staff members and partners are routinely asking the right questions, sharpening both the theoretical framework (e.g., “Why are we doing this? What do we hope to achieve?”) and the daily routines of program implementation (e.g., “What is our baseline? Do we expect to see change?”). Despite the challenges, building data skills remains essential to improving the practice of community development and most importantly, the lives of the residents community development programs seek to impact.
Integrating a data mindset and skill set into LISC Chicago is analogous to realizing as a parent, on high school graduation day, that you should have photographed your child more often when she was entering kindergarten. It is challenging to attempt to build this capacity and retrofit the data models into a methodology that LISC has worked on for 15 years. Also, in the neighborhoods where LISC works, there are some things that data cannot help and data cannot do. Innovative practices will likely not have a baseline or evidence base, and serendipitous outcomes will not be captured by pre- and post-treatment surveys. Like parenting, community development is complex. Even as we try to capture the decisive moments, we must leave room for the unpredictable, the messy, and the surprise endings.
 Capital absorption capacity represents “the ability of communities to make effective use of different forms of capital to provide needed goods and services to underserved communities,” according to the Living Cities Integration Initiative, which identified gaps in such capacity as barriers to neighborhood improvement. See more at https://www.livingcities.org/work/capital-absorption.
 D. Greenberg, et al., “Creating a Platform for Sustained Neighborhood Improvement,” MDRC, February 2010.
 K. Mossberger and C. Tolbert, “Digital Excellence in Chicago: A Citywide View of Technology Use,” (City of Chicago Department of Innovation and Technology, July 2009). www.smartcommunitieschicago.org/uploads/smartchicago/documents/digital_excellence_mossberger_study.pdf.
 C. Tolbert, K. Mossberger, and C. Anderson, “Measuring Change in Internet Use and Broadband Adoption: Comparing BTOP Smart Communities and Other Chicago Neighborhoods,” (University of Iowa and University of Illinois at Chicago, 2012). http://www.brookings.edu/blogs/techtank/posts/2014/10/27-chicago-smart-neighborhoods.