Theory, practice, and how they’re measured: MAAC preps for constant changeMarc Schultz
This year, the Building Community Network program – a partnership among GCN, The Home Depot Foundation, and hundreds of select Atlanta-area nonprofits – has focused on tactics and strategies for leading in times of change. Over the course of four events, leaders from over 100 network partners have come together to learn, practice, and discuss change management with the organizations and experts leading the way.
In January, Multi-Agency Alliance for Children (MAAC) ED Heather Rowles addressed the BCN community at an event hosted by Trees Atlanta, presenting her organization’s solution to the ongoing challenge of communicating the organization’s worth to the government appointees who control the organization’s contract with the State, and who change regularly. The presentation from Rowles chronicled MAAC’s journey to discover the results measures that best related the story of their impact, and how to track them across the entire organization.
A unique provider
MAAC provides services that are unique in the state, especially for the child-welfare space, bridging 10 different Georgia agencies whose efforts support youth in the foster care system. “Our staff is dedicated to making sure that what kids need, they get – as opposed to whatever happens to be available,” said Rowles. “We come up with all kinds of creative solutions in order for kids to be successful.”
Though their service contract has been unchanged since 1998, it’s not something the nonprofit can afford to take for granted.
Though their service contract, which MAAC maintains with Georgia’s Division of Family and Children’s Services (DFCS), has been unchanged since 1998, it’s not something the nonprofit can afford to take for granted. “Every time there’s a change in leadership at DFCS – every time we have a new director, Human Services Commissioner, or Governor – I have to be upfront in selling our program,” said Rowles. That requires making the case anew for MAAC’s value to the state.
To keep up with changes in state leadership and increasingly rigorous evaluation standards, Rowles knew she needed to demonstrate exactly how MAAC’s work was impacting youth in foster care, and what kid of return the state was getting on its investment.
Building the case, part 1: Theory of change
The potential of MAAC’s work is illustrated by their Theory of Change, a one-page document that delineates who MAAC serves, how MAAC serves, the direct results of that service, and the ultimate goals of their work, including the ways that their work affects Georgia’s child welfare system as a whole.
Rowles and her team were introduced to the Theory of Change design process through Leading for Impact, a results-focused capacity-building program offered by Bridgespan and GCN. They received further support through the Annie E. Casey Foundation’s Results Count program, a similar leadership development initiative that put Rowles and another MAAC staffer on a team with a leader from Georgia Dept. of Education, a leader at DFCS, and a young person who had experienced foster care. Working alongside these stakeholders also helped MAAC determine their most vital outcomes – the kind that put significant weight behind their answer to the fundamental question, “Are kids better off because of what MAAC is doing?”
Altogether, it took the MAAC team 6 months, and 18 versions, to produce their final Theory.
In addition to establishing outcomes related to child welfare – those that measure placement stability, provider effectiveness, youth self-efficacy, and more – they also needed measures that demonstrated a statewide return on investment. “How can we show that the work we're doing is truly making a difference on the entire system, saving the government money?” asked Rowles. “What does that investment really get them?”
Altogether, it took the MAAC team 6 months, and 18 versions, to produce their final Theory. “We vetted it with our partner agencies, our staff, our external partners, DFCS, everybody,” said Rowles. “We wanted to make sure we got it right.”
Building the case, part 2: Dashboards
With the right outcomes encoded in their Theory of Change, it was time to put their efforts on display. That meant creating specific measures for quantifying progress toward those outcomes, and dashboards where they could document that progress.
Six months ago, MAAC contracted GCN Senior Consultant and EVP Kathy Keeley to help them in this next phase. Though it has also been another time-consuming project, Rowles reported that the results are worth it: “Every word is very meaningful.”
To get to this point, MAAC found (somewhat counter-intuitively) that they actually needed to reduce the amount of data they were tracking. “We discovered in this process that we were measuring too much,” said Rowles. “All we needed were the things that really let us know whether or not we were making a difference.”
MAAC found (somewhat counter-intuitively) that they actually needed to reduce the amount of data that they were tracking.
For instance, their Crisis Continuum Program (CCP) is designed to solve the problem of too few foster homes, which leads children in the system to be frequently moved, and even to be kept in hotel rooms – a disruptive and often traumatizing process. The main indicator for program success is a quality called “Placement Stability.” Measuring that quality, they determined, is a matter of four factors:
The percentage of youth completing their CCP individual action plan within 30 days
The percentage of youth in appropriate, safe, and stable placement upon exiting the program
The percentage of youth served who transition from temporary arrangements to safe, stable placements; and
The percentage of youth with zero disruptions in the first 90 days of placement.
The program dashboard where they keep track of those critical outcomes measures – the difference made – also includes output measures, tracking the work itself: the number of youth served, number of planning meetings, the average number of days youth spend in the program, etc. Altogether, said Rowles, this dashboard answers the question, “How much are we doing through the Crisis Continuum Program, and how well are we doing it?”
With a dashboard set up for each of their programs, MAAC can produce a complete picture of the progress they’ve made against the ultimate goals set out in their Theory of Change. To do that, MAAC just pulls the bottom-line data from each dashboard into an all-encompassing “Results Dashboard,” listing each of their ultimate goals, the measure they’re using to quantify them, and the programs involved.
This Results Dashboard, Rowles noted, is shared with the people who make them happen as well as the people who fund them: “The entire organization sees these dashboards, and so do our partner agencies,” said Rowles. It’s all a part of creating a data-driven culture, beginning at the top and including everyone on the front lines. “Part of our success has been because it’s really embedded in the work we all do.”
GCN CEO Karen Beavor followed up the presentation by emphasizing the ripple effects of this new data-first approach. “These measures will drive how you’re now going to think about fundraising, about creating impact, and carrying the message to donors and investors,” said Beavor. “Service model transformation, revenue model transformation, data transformation, these things are intertwined. Whichever end you start at, it’s going to lead you to the others.”
Marc Schultz is communications editor at GCN.