Moneyball and the Art of Transparency and Accountability

On a quick three-day work trip to Prague this week (to help run a Global Integrity Dialogue workshop with the Czech Ministry of Interior and US Embassy), I finally had a chance to read Michael Lewis’ classic Moneyball: The Art of Winning an Unfair Game. Moneyball tells the story of how the Oakland Athletics baseball team managed to win so many games in the early-2000s despite having one of American baseball’s smallest payrolls and, subsequently, so few of its star players.

The short answer that Lewis brilliantly chronicles: the team embraced a hard-nosed approach to relying on new, more accurate statistics of players’ performance rather than on old-time baseball gut “feel” to evaluate players. Those insights allowed the team to buy up undervalued, less-sexy players on the cheap, who, it turned out, were also far more likely to contribute to runs scored and therefore games won.

The game-changing statistic embraced by the A’s and ignored by much of baseball was on-base percentage. Far more important likely to determine whether runs would be scored (and thus games actually won) than any other statistic, on-base percentage become the success metric around which the team built its personnel strategy.

So this got me thinking: what’s the transparency and accountability movement’s Moneyball statistic? Do we have a performance measure that we know matters more than others when it comes to successful transparency and accountability outcomes?

The short answer is no, I believe, for unsurprising reasons. Baseball lends itself more easily to this analytical framework because it has an easily measureable outcome (dependent variable) against which various inputs (independent variables) can be measured in terms of their efficacy: wins. A win is a win, a loss is a loss, and if you dig deep enough (as the A’s and others did) you eventually uncover inputs that tend to drive outcomes more regularly than others.

As someone who has spent the better part of the past decade working towards better ways of measuring anti-corruption and accountability mechanisms, this is sobering stuff. We are way behind baseball, despite baseball having an easier time of it thanks to their Win/Loss column. What is a “win” in transparency and accountability work? That’s a pretty tough question to answer. Since we can’t often answer it with precision, we tend to fall back on vague euphemisms: “reducing corruption” or “increasing trust in government.”

But these aren’t real, measureable "wins."  They are more old-time baseball “feel” than anything else.  If you’ve been around anti-corruption long enough, the unspoken mantra goes, you know what a “good” project or policy initiative is when you see it.  Just like a old-fashioned baseball scout “knows” a good high school prospect when he seems him.

Except that the A’s proved that the old-time scouts were often dead wrong.

Does the transparency and accountability movement have an Oakland A’s? The crop of economists experimenting with randomized control trials are certainly helping to build some evidence around what works and what doesn’t with a certain scientific precision. Ben Olken’s seminal work in Indonesia has helped to pave the way for others, many of whom are affiliated with MIT’s Jameel Poverty Action Lab. Their efforts are time-consuming and often difficult to replicate in other countries, but they really matter.

What RCTs have going for them is their ability to precisely define what a “win” is in the transparency and accountability context: dollars lost or saved; medicines on the shelf or not; textbooks on the schoolroom shelf or not. These are measurable, quantifiable outcomes whose improvement (or lack thereof) can be tracked based on changes to the inputs (transparency and accountability interventions). Once in a while, when we’re lucky, we can isolate the causal relationship and say, “We know that if you put cameras in a classroom in this district, more teachers will show up to teach their students.”

So if we know that RCTs and similar approaches are helpful in that their design forces us to more clearly define the transparency and accountability “win,” why aren’t those approaches multiplying more rapidly?

A partial answer, in my view, is that much of the transparency and accountability community of practice is geared towards working at a level of government that is quite often delinked from easily measureable “wins.” We (including Global Integrity) typically work with national governments, multilateral aid agencies, and non-governmental groups based in capital. Their work, even when effective, is often hazy enough to make it difficult to identify the real-life “wins.” Good public policy, “awareness raising,” and new forms of institutional management are not the “wins” we are interested in measuring, ultimately. We theorize that they can contribute positively to whether drug stocks appear on the clinic shelf because theft and bribery have been reduced, but we’re talking about “outputs” versus “outcomes.” The transparency and accountability movement is desperately in need of more measurable outcomes. We need to find our “wins.”

Where to identify more “wins?” If nothing else, we need as a community practice to continue pushing aggressively at the local and sector levels in countries; if this means scaling back national programs and campaigns, that’s a trade-off I am very happy to make. “Going local” gives us a much better chance of designing our work in a way that identifies “wins” in a concrete way. If that work turns out to be successful, and if we can document some of the causal pathways, we can potentially scale the successes both horizontally and vertically within and across countries. In lay terms: let’s figure out what it takes to reduce corruption in a way that leads to drug stocks showing up more regularly in a single village clinic before we launch another high-profile international campaign to “stamp out corruption” in the global pharmaceutical sector.

Far too much of our work starts (and ends) at the country and even international levels, and a whole lot of it is ineffective (see my continued rants on national anti-corruption commissions). Let’s instead look for opportunities to rigorously test and document our successes and failures at the local level first, and then explore whether they can travel. We need to discover and define our “wins” and begin innovating from the bottom up.

— Nathaniel Heller

— image credit: mrjoro