The title hooked me immediately. I’m an avid follower of the @TEDTalks Twitter page, but I don’t have time to watch every talk. But when I saw one with the title ‘Want a more innovative company? Hire more women’, I wanted to hit play instantly. My most-cited paper is on corporate culture, and just one month ago I’d appointed the first female editors to the board of the Review of Finance, a leading academic journal — the first appointments I’d made after taking over as Managing Editor earlier that year.
The speaker described how her research team investigated the link between diversity and innovation. Like all good TED talks, she takes you on a journey, claiming they started out with skepticism on whether they’d find anything at all:
To put it mildly, we were not optimistic. The most skeptical person on the team thought, or saw a real possibility, that we would find nothing at all. Most of the team was rather on the cautious side, so we landed all together at ‘only if’, meaning that we might find some kind of link between innovation and diversity, but not across the board — rather only if certain criteria are met, for example leadership style, very open leadership style that allowed people to speak up freely and safely and contribute.
Their boldest hope was to find that diversity works in specific situations. But as the speaker continues her journey, there’s a twist which makes her findings even more striking:
A couple of months later, the data came in, and the results convinced the most skeptical amongst us. The answer was a clear yes, no ifs, no buts. The data in our sample showed that more diverse companies are simply more innovative, period.
This was music to my ears. But I wanted to make sure that I’d understood the study correctly, and so I listened to the talk again — this time trying to step outside the story and focus on the evidence. Now, my ears pricked up at something else.
To measure diversity, we looked at six different factors: country of origin, age and gender, amongst others.
In sharp contrast to the title, which focused exclusively on gender diversity, the researchers actually measured six different dimensions of diversity. Even if there was a cast-iron link to innovation, you had no way of knowing whether it was gender diversity, or any of the other dimensions, that was behind it. The title was no more valid than ‘Want a more innovative company? Hire older people’ or ‘Want a more innovative company? Hire younger people’ — but such titles would have likely led to far fewer views.
And the problems didn’t just stop at their measure of diversity, but also of innovation. The most widely accepted yardstick is the number of patents a company produces, as a patent is only granted if something is truly innovative. If you are about quality, not just quantity, you could measure the number of citations to a patent, the market value of a patent, or how distinct it is from a company’s existing patents.
But the team did something quite different. They measured the share of a company’s revenues that came from new products developed over the past three years. That fraction could capture many things other than innovation. It could be high because you’re not innovative – the products you launched four years ago weren’t that ground-breaking, so they fell away fast and are no longer generating sales. Or maybe you did a bad job of marketing them, or a new CEO came in and decided to change direction. Alternatively, you could be generating sales from new products even if they’re not that innovative — they could be a small variation on an existing theme. Even if the 15th edition of a textbook is little different from the 14th, most people will get the latest one.
Perhaps due to viewer complaints, TED have since changed the title of the talk to ‘How diversity makes teams more innovative’, which addresses the first problem. But it doesn’t solve the second, which is part of the same concern — when a study parades a relationship between two things, check and see what it actually measured. If it’s something different from what the authors claim, then a statement is not fact. (And the new title doesn’t address a separate problem, that there’s correlation but not causation, as covered in the section on why data is not evidence.)
