Data- and Experiment-Driven Policy Development

In God we trust. All others: bring data. — William E. Deming

Developing countries face public-policy challenges no less difficult — but in many ways with far more pressing urgency — than their more developed counterparts in the west. These challenges include the provision of basic public services — consistently, reliably and efficiently — across the urban-rural divide, as well as across other dimensions of societal differentiation. These challenges also include the problems of sustainable development, that is, achieving improvements in meeting economic, environmental and social needs without creating imbalances that imperil future generations.

Successive waves of academics, consultants and experts have travelled eastward and southward to developing countries with the promise of a solution. Sometimes the offered solution is described as being of well-established efficacy, having been proven by well-established practice in western societies. Sometimes the offered solution is more novel, promising revolutionary transformative effects.

Examples, which have been accumulating for many decades, range from IMF conditionality (on government borrowing and spending, on tariffs and market access, financial- and private-sector reform conditions, sector-specific structural reform conditions, etc.) through to public-sector policies, industrial policies, market-reform policies, large-scale infrastructure projects, and even agricultural, land and health policies.

Social science, broadly conceived — to the extent that it can be considered a science — differs from natural science in one key respect: the unit of observation is either a social aggregate or an individual human being who cannot be completely divorced from cultural and social influences. Culture and social institutions differ from place to place on the globe, as well as from one epoch to the next in human history.

Policies and management practices that work under one organisational culture with its associated social institutions do not necessarily work at all under different institutions and culture.

This ‘elephant in the room’ — which can be traced to a failure to account for David Hume’s famed ‘problem of induction’ — lies at the heart of many western-inspired grand policy-experiment failures. Indeed, it is the problem that must be faced in any communication of recommendations or ‘best practice’ from the western world to the developing world.

But the problem not only applies to transporting western solutions to developing countries. It applies to all attempts to transport solutions from one place to another. No less so when there is an underlying gradient of power or development: from colonial power to colony, from developed world to developing world, from centre to periphery, from urban to rural.

The 2009 Nobel-Prize winner Elinor Ostrom dedicated her life’s work to generating a diverse and wide-ranging body of evidence on how communities may resolve the tragedy of the commons. She was convinced — and succeeded in convincing the world — that there are, in fact, many ways to overcome the tragedy of the commons, but that centrally imposed solutions do not generally work. She and her collaborators created a large body of evidence on examples of local communities developing their own successful solutions to the tragedy of the commons.

The key to such successful solutions, Elinor Ostrom has shown us, is that they must arise from the community itself. When they do so, they take account of the local geography, the local ecology, the skills and interests of the local population, the local culture, the local customs, and the local institutions.

The public-policy challenges of developing countries cannot be solved by transplanting western solutions. It may seem that the Six-Sigma approach to process improvement can find ready application — and deliver large efficiency improvements — when applied to public service provision in developing countries. But without fundamental adaptation to the developing-country context, there is a great risk of it remaining of questionable efficacy.

Fortunately, both business and social science have been undergoing a radical transformation in the last 20 years that has fundamentally changed the way in which policies and solutions are developed. ‘Design’ — whether it be product design, service design, process design, or policy design — has been supplanted by a data-driven model.

Rather than tasking an expert with designing a solution, the data-driven model requires every concept and feature to arise not from theory or ideology, but from a reliable pattern in data, whether the data happen to have been collected passively or actively through an experiment.

For instance, the particular hue and saturation of blue that Google renders its links with was not conceived by a designer. Instead, Google ran thousands of experiments to find out what colour and hue leads to the most clicks. In fact, it is Google’s policy that any proposed feature cannot be incorporated into a Google product without it having delivered a statistically significant improvement in field-experiment testing (called A/B testing in computer science).

In the UK, the Behavioural Insights Team (BIT) — a.k.a. the ‘nudge unit’ — has been advising the Cabinet Office and other government departments since 2010. It designs field experiments to test the extent to which small changes to the presentation format and framing of policies and policy materials — i.e. changes which exploit insights from behavioural economics and psychology — successfully improve the attainment of policy objectives.

The BIT has gained wide recognition for its success in measurably increasing performance across the public sector, for instance increasing tax payment and collection rates, increasing fine payment rates, reducing medical prescription errors, increasing organ donation rates, and increasing households’ installation of loft insulation. The UK BIT is now being emulated in many different countries.

The disciplines of political science, development economics, and public health have also embraced the use of randomised controlled field experiments for determining the effectiveness of a wide array of different policy measures and interventions. Especially in developing countries, field experiments have become the gold standard for evaluating policy measures. Rather than committing a nation to a big policy revolution ending in a failed grand experiment, with randomised controlled field experiments there is no head-over-heels commitment to a particular solution — until and not unless a particular intervention proves itself to be efficacious in the local institutional and cultural context.

In the course of carrying out its functions, the nation state generates large quantities of administrative data. Through Tim Berners-Lee’s successful championing of ‘open data’ — i.e. making available on the internet fine-grained location-specific public services data — the performance of UK public services can now be studied by both academics and the public. All government departments in the UK are expected to publish their administrative data that does not violate data-protection regulations, unless there are other extenuating circumstances. As a result of the open-data initiative, there is now much more transparency in the delivery of public services. Consequently, data-driven policy initiatives are now possible on a much more democratised basis.

Mobile telephony and internet-enabled systems and devices produce even larger quantities of unstructured data. The relatively new discipline of ‘data science’ has emerged around the application of machine-learning techniques to such unstructured ‘big data’. The machine-learning tools of data science are being used to detect and validate patterns which can be exploited to enhance organisational effectiveness. Data Scientists are in high demand, both in the commercial sphere as well as in government institutions, and accordingly command premium salaries.

Data- and experiment-driven policy development offers a way to avoid the fundamental mistake of blindly applying developed-country solutions to a developing-country context. Whereas Elinor Ostrom showed that communities can reach their own solutions, given stable conditions and sufficient time, a data- and experiment-driven approach gives local conditions their due voice and shortens the timeframe required for the locally adapted solution to emerge.

At this time in history we can acknowledge the harmful folly of unquestioningly embracing grand-solution policies from abroad.

Fortunately, we also have access to data- and experiment-driven techniques that allow policy proposals to be evaluated in the context of local institutional and cultural factors. With the tools of statistics, data science, and experimental science, developing-country citizens can develop and refine their own policy solutions — as they should.

This article provides an overview of the lecture ‘Data- and Experiment-Driven Policy Development’ given by the author at De La Salle University (13.3.2017) and Ateneo De Manila University (14.3.2017) as part of the Great Lecture Series sponsored by the British Embassy in Manila.