by Michael Luca and Max H. Bazerman
MIT PRESS, 2021, 232 PP. PAPERBACK, $19.95; EBOOK, $19.95 ISBN: 978-0-262-54227-2
How did experiments, once cloistered in laboratories, become a contemporary lever wielded widely in both public and private sectors today? Michael Luca and Max H. Bazerman trace this lineage in their book, The Power of Experiments: Decision Making in a Data-Driven World. The book is written for a practical audience envisioned to be managers and other businessfolk interested in learning the power of experiments (how experiments have come to influence action in contemporary commerce) and frameworks for further application (i.e., how readers might take what they learn from this book and apply it in addressing their own business problems).
The book is organized into three parts. Part 1 provides the background of experiments, including a primer on the disciplinary history of experiments and behavioral science’s roots in experimental psychology and experimental economics. Part 2 explores the popularity of experiments at tech companies, reviewing a number of case studies from companies such as Airbnb, eBay, Booking.com, Uber, and more. Part 3 examines the use of experiments in a number of “social good” sectors beyond tech, including health, education, and finance. The book concludes with some brief parting remarks on the ethics of experiments and ends with a number of key take-aways.
From the jump, readers are immersed in a story of how the influential UK government’s Behavioural Insights Team (BIT) began in 2010. This first-of-its-kind, interdisciplinary group was tasked with using behavioral science to improve government and policy. BIT would later become known as the “Nudge Unit.” Its first task was to help the UK government collect unpaid taxes. After years of planning and debate, BIT was approved to run an experiment that added one sentence to the form letter sent to people who were late in paying their taxes: “By now, 9 out of 10 people in your town have paid their taxes.” This simple addition increased tax payments by 2 percent. With this, the Nudge Unit blasted off on what would be a trajectory of wild success.
The book then goes on to give the readers a working vocabulary of experimental methods by laying out the anatomy of an experiment—control group, treatment group(s), independent variable, dependent variable, average treatment effect—in the context of the BIT tax example. Continuing with a history of experiments, this genealogy is at times oddly sweeping (e.g., referencing the biblical book of Daniel) and at other times surprisingly amiss (e.g., failing to mention modern atrocities such as the Tuskegee Syphilis Study), [End Page 97] though it does introduce the reader to crucial issues such as selection bias and offer some examples of how this bias warps medical experiments and resultant medical knowledge.
Continuing with its primer, the book provides readers an introduction to the behavioral sciences. Far from a dry history, the prose is peppered with maxims for practitioners and lessons taken from both experimental psychology and experimental economics. From the principles of Skinner and Pavlov to the horrors of Milgram and Eichmann, the chapter advises the reader, “When devising an experiment, organizations need to consider the potential stress and harm that the experiment could impose on participants. [I]t’s important to have explicit discussion of the potential risks and harm involved” (23). Exactly how organizations ought to go about having these discussions or who should have a seat at the table is not clear. Further advice drawn from debates among experimental economists tells the reader: “There is no single perfect way to run an experiment. [It is important to think] carefully about the question you are trying to answer before designing an experiment” (31). The reader will no doubt find such maxims important and compelling, even if their actionability remains harder to discern.
The book offers case after case in which experiments have been used to drive decision-making in real-world settings, delving deep into their explosion in the tech sector. There, experiments are often called “A/B testing”; it is cheap to tweak software conditions, thus making the cost of manipulating customers low. Beyond running specific experiments, the book highlights the need for experimental infrastructures, organizational change, training, and social scaffolding needed within an organization to cultivate an experimental mindset there.
This experimental mindset is something readers are encouraged to take on and bring into their everyday work practice, framed as a form of humility: “Good leaders have enough humility to admit what they don’t know, identify the best options in an uncertain world, and experiment” (176). All day long, decisions big, small, and in-between are made in meetings where people take leaps of faith but act with confidence—and, at times, act with hubris. Advocating, then, for a humble mindset seems radical and even necessary in our contemporary context where headlines are filled with corporate decisions that contribute to growing inequalities.
But where the book falls short is in its treatment of the ethics of experiments. The book’s penultimate chapter on the topic is a mere four pages. It introduces another concept, experiment aversion, only to cursorily shun it as “people’s fear of being guinea pigs in an organization’s experiments” (175). Instead of engaging with important ethical issues in experiments (e.g., problems of consent or the impact of deception without debriefing), this chapter argues that the experimental mindset creates a “moral imperative to experiment” (175).
The proliferation of experimental methods across a number of public and private sectors raises important and perhaps disturbing questions around what it means to participate and be a participant in public life today. What does consent mean if there is no alternative? How do concepts of accuracy or truth hold up if we are all interacting with slightly different versions of the world as we move through experiments? If we return to the BIT tax example, is the sentence about “9 out of 10 people” true? What does it mean to receive a letter from the government saying so? These are just some of the questions that come up to quickly cloud any black-and-white claims on a “moral imperative to experiment.”
Still, this monograph offers a lot to its readers and would do well in an introductory survey class, especially since it offers a concise background for the reader unfamiliar with [End Page 98] experiments and their long arm in contemporary sociotechnical systems. Given its lack of depth on ethical considerations, it could be paired with case studies from the works of Joy Buolamwini, Safiya Umoja Noble, or Latanya Sweeney, for example, to add needed nuance to these pressing matters of concern. [End Page 99]
—Christine T. Wolf, Thomson Reuters Corporation