Experimenting on WEIRD people

27 Oct 2023 | Evidence is not proof

One of my research fields in behavioural economics, which suggests that people don’t always act in perfectly rational ways. A famous experiment in this field is known as the Ultimatum Game. There are two players, a ‘proposer’ whom we’ll call Amelia, and a ‘responder’, Bilal. Amelia is given £10 and is then asked to suggest how to share it with Bilal. If Bilal accepts the split, both players receive Amelia’s proposal; if he rejects it, both get nothing. It’s rational for Bilal to accept any offer of at least £0.01, because anything is better than zero. Knowing this, Amelia should only suggest £0.01, keeping £9.99 to herself.

But experiments show that Bilal typically rejects any split under £3. He views such a division as unfair and rejects it to punish Amelia, even though it means he also gets zero. Fairness seems to matter even more than putting food on the table.

The first findings were from an experiment on University of Cologne economics students in 1978 (that was published in 1982). One concern with experiments is that they might be unique to a specific setting — students, rather than working professions, and Cologne instead of Colorado. Indeed, the ‘replication crisis’ in social science has arisen because some landmark findings have been overturned in other settings. But the Ultimatum Game has been widely replicated. A study co-authored by Nobel Laureate Alvin Roth repeated it on cities as far afield as Jerusalem, Ljubljana, Pittsburgh, and Tokyo. It’s also been documented in five-year-old kids, and even chimpanzees.

Yet as I dug deeper into this topic for a behavioural economics paper I was writing, I discovered the results weren’t actually so universal. Psychologist Joseph Henrich tested the Ultimatum Game on the Machiguenga people of the Peruvian Amazon. Proposers made miserly offers, and responders gratefully accepted pitiful amounts without viewing them as unfair. As Henrich explained, “Rather than viewing themselves as being ‘screwed’ by the proposer, they seemed to feel it was just bad luck that they were responders, and not proposers.”

Quite why Machiguengans play the Ultimatum Game differently from others is unclear. It’s tempting to dismiss them as being selfish for making low offers, but that would be imposing our own notions of fairness. Machiguengans might equally call it selfish to think you deserve a share of someone else’s good fortune, and to be so jealous of it that you’ll punish them by rejecting a split. Regardless of what causes these differences, they exist. Henrich later wrote  a book pointing out how studies of human behaviour focus on WEIRD subjects — those who are Western, Educated, Industrialized, Rich and Democratic — and their results may not extend to the rest of the world.

In the Ultimatum Game, Amelia gets the £10 as a pure windfall — it drops from the sky like manna from heaven. But what if she had to work for it? Other researchers ran an experiment where two kids fished for magnetic cubes out of different containers. The game was rigged so that some cubes were less magnetic and fell off the rods more easily, but the children didn’t know this. As a result, several pairs ended up with uneven spoils. The catch was combined, and the kids were asked to suggest how to split them. German children thought it’s fair for the one who caught the most to get the lion’s share. But to kids from the #Akhoe Hai//om community in Namibia, fairness meant an equal split — regardless of how much each child contributed.

This is an example of why evidence is not proof. As I wrote in Grow the Pie, “A proof is universal. When Archimedes showed that the area of a circle is pi times the square of its radius, he proved this not just for circles in Ancient Greece in the third century BC, but for circles in Modern Greece today and for circles throughout the world. But evidence may only apply to the country or industry in which it was gathered.”

Even if you have the world’s most rigorous evidence, the results might not extend to other settings.

Inside the Ivory Tower

Inside the Ivory Tower

In May Contain Lies, I highlight the value of academic research. While it's far from perfect, it can be more reliable than practitioner studies for a number of reasons: Its goal is scientific inquiry, rather than advocacy of a pre-existing position or releasing findings to improve a company's image. It's conducted by those with expertise in conducting scientific research. Papers published in top scientific journals are peer-reviewed, which helpsimprove their accuracy. However, authors, journalists, and practitioners will sometimes cite research as if it bears the hallmark ...
Does only 2% of VC funding go to female founders?

Does only 2% of VC funding go to female founders?

A widely quoted statistic is that only 2% of VC funding goes to female founders. For example, this Forbes article highlights that "only 2% of all VC funding goes to women-led startups" and asks "Why is only 2% of VC funding going to female founders"? If true, this statistic is substantial underrepresentation and needs to be urgently addressed. However, it's problematic for several reasons. 1. The Statistic Ignores Diverse Teams The 2% statistic actually refers to companies founded solely by women. It ignores diverse companies founded by both men and women. This is strange, because ...
An unhealthy obsession with organisational health

An unhealthy obsession with organisational health

Two leading asset management firms drew my attention to the McKinsey Organizational Health Index as a potential tool to evaluate a company. A book, "Beyond Performance 2.0: A Proven Approach to Leading Large-Scale Change", written by two McKinsey partners, claimed that companies with high scores on this Index trounced their unhealthy peers along a range of performance measures. For example, their shareholder returns were three times as high. But as I wrote in an earlier post, rather than being more impressed by big numbers, we should be more sceptical. If it were really possible to ...