Last week, Bill Gates’ 2013 Annual Letter highlighted the power of data and measurement to help lift the world’s most needy up from poverty. I heard about it from several tweets asking me if he had read “Measuring the Networked Nonprofit” and colleague, Marion Conway, wrote a great summary of the letter which was also available in an interactive format on the web.
There are many takeaways from his letter, but I wanted to identify a few inspiring quotes here.
1. Define A Clear Goal for Success and Pick the Right KPI
“In the past year I have been struck again and again by how important measurement is to improving the human condition. You can achieve amazing progress if you set a clear goal and find a measure that will drive progress toward that goal—in a feedback loop. This may seem pretty basic, but it is amazing tome how often it is not done and how hard it is to get right. Given a goal, you decide on what key variable you need to change to achieve it—the same way a business picks objectives for inside the company like customer satisfaction—and develop a plan for change and a way of measuring the change. You use the measurement as feedback to make adjustments. I think a lot of efforts fail because they don’t focus on the right measure or they don’t invest enough in doing it accurately.”
2. Use Measurement To Improve Results
“Yet while measurement is critical to making progress in global health, it’s very hard to do well. You have to measure accurately, as well as create an environment where problems can be discussed openly so you can effectively evaluate what’s working and what’s not.”
This is a very important point: You have the apply the data to decision-making to improve your program’s results. Gates is talking about how an NGO’s internal culture needs to be open to learning what isn’t working as well well as what is working. This gets at one of the areas of measurement that interests me the most – the intersection between measurement discipline and organizational culture.
The Open the Echo Chamber blog points how difficult this in practice:
But none of this matters if all of the institutional pressures run against hearing bad news. Right now, donors simply cannot tolerate bad news, even in the name of learning. Certainly, there are lots of people within the donor agencies that are working hard on finding ways to better evaluate and learn from existing and past programs, but these folks are going to be limited in their impact as long as agencies such as USAID answer to legislators that seem ready to declare any misstep a waste of taxpayer money, and therefore a reason to cut the aid budget…so how can they talk about failure?
The blog goes on to make a proposal to Gates about how to change the culture:
Please round up a bunch of venture capitalists. Not the nice socially-responsible ones (who could be dismissed as bleeding-heart lefties or something of the sort), the real red-in-tooth-and-claw types. Bring them over to DC, and parade out these enormously wealthy, successful (by economic standards, at least) people, and have them explain to Congress how they make their money. Have them explain how they got rich failing on eight investments out of ten, because the last two investments more than paid for the cost of the eight failures. Have them explain how failure is a key part of learning, of success, and how sometimes failure isn’t the fault of the investor or donor – sometimes it is just bad luck. Finally, see if anyone is interested in taking a back-of-the-envelope shot at calculating how much impact is lost due to risk-averse programming at USAID (or any other donor, really).
3. It’s About Sense-Making, Not Tools
Gates shares many examples from their work. Like Marion Conway, I was struck by this photo of him pointing to data handwritten on the walls = tracking immunization records from a rural health clinic in Ethiopia.
Stories of progress like this underscore the importance of setting goals and measuring progress toward them. A decade ago, there was no official record of a child’s birth or death in rural Ethiopia. At the Germana Gale Health Post, I saw charts of immunizations, malaria cases, and other health data plastered to walls. Each indicator had an annual target and a quarterly target. All this information goes into a government information system to generate regular reports. Government officials meet every two months to go over the reports to see where things are working and to take action in places where they aren’t.
Ethiopia’s recent effort to monitor the progress of its immunization program is a good example of learning from data and—the hardest part—using data to improve delivery of the right solutions. A recent national survey of Ethiopia’s vaccination coverage reported vastly different results fromthe government’s own estimates. Ethiopia could have ignored this conflict and reported the most favorable data. Instead, it brought in independent experts to understand why the measurements were so different. They commissioned a detailed independent survey that pinpointed geographic pockets of very high coverage—and very low coverage. The government is now working to develop better plans for the poorer performing regions.
It illustrates that an idea that Katie Paine and I emphasize in our book, “Measuring the Networked Nonprofit.” It isn’t about the tools. It is isn’t about collecting hundreds or thousands of data points. It is about defining the success, collecting the right data points, and making sense of it to improve outcomes. Simple, but hard to accomplish.
Is your nonprofit effective at using measurement? Does your organization have a culture where it discusses and embraces failure for learning? Are more focused on applying the data than the tools to collect it?
