Want a more innovative conclusion? Innovate the conclusion

2 Jan 2024 | A statement is not fact, Diversity

The title hooked me immediately. I’m an avid follower of the @TEDTalks Twitter page, but I don’t have time to watch every talk. But when I saw one with the title ‘Want a more innovative company? Hire more women’, I wanted to hit play instantly. My most-cited paper is on corporate culture, and just one month ago I’d appointed the first female editors to the board of the Review of Finance, a leading academic journal — the first appointments I’d made after taking over as Managing Editor earlier that year.

The speaker described how her research team investigated the link between diversity and innovation. Like all good TED talks, she takes you on a journey, claiming they started out with skepticism on whether they’d find anything at all:

To put it mildly, we were not optimistic. The most skeptical person on the team thought, or saw a real possibility, that we would find nothing at all. Most of the team was rather on the cautious side, so we landed all together at ‘only if’, meaning that we might find some kind of link between innovation and diversity, but not across the board — rather only if certain criteria are met, for example leadership style, very open leadership style that allowed people to speak up freely and safely and contribute.

Their boldest hope was to find that diversity works in specific situations. But as the speaker continues her journey, there’s a twist which makes her findings even more striking:

A couple of months later, the data came in, and the results convinced the most skeptical amongst us. The answer was a clear yes, no ifs, no buts. The data in our sample showed that more diverse companies are simply more innovative, period.

This was music to my ears. But I wanted to make sure that I’d understood the study correctly, and so I listened to the talk again — this time trying to step outside the story and focus on the evidence. Now, my ears pricked up at something else.

To measure diversity, we looked at six different factors: country of origin, age and gender, amongst others.

In sharp contrast to the title, which focused exclusively on gender diversity, the researchers actually measured six different dimensions of diversity. Even if there was a cast-iron link to innovation, you had no way of knowing whether it was gender diversity, or any of the other dimensions, that was behind it. The title was no more valid than ‘Want a more innovative company? Hire older people’ or ‘Want a more innovative company? Hire younger people’ — but such titles would have likely led to far fewer views.

And the problems didn’t just stop at their measure of diversity, but also of innovation. The most widely accepted yardstick is the number of patents a company produces, as a patent is only granted if something is truly innovative. If you are about quality, not just quantity, you could measure the number of citations to a patent, the market value of a patent, or how distinct it is from a company’s existing patents.

But the team did something quite different. They measured the share of a company’s revenues that came from new products developed over the past three years. That fraction could capture many things other than innovation. It could be high because you’re not innovative – the products you launched four years ago weren’t that ground-breaking, so they fell away fast and are no longer generating sales. Or maybe you did a bad job of marketing them, or a new CEO came in and decided to change direction. Alternatively, you could be generating sales from new products even if they’re not that innovative — they could be a small variation on an existing theme. Even if the 15th edition of a textbook is little different from the 14th, most people will get the latest one.

Perhaps due to viewer complaints, TED have since changed the title of the talk to ‘How diversity makes teams more innovative’, which addresses the first problem. But it doesn’t solve the second, which is part of the same concern — when a study parades a relationship between two things, check and see what it actually measured. If it’s something different from what the authors claim, then a statement is not fact. (And the new title doesn’t address a separate problem, that there’s correlation but not causation, as covered in the section on why data is not evidence.)

Inside the Ivory Tower

Inside the Ivory Tower

In May Contain Lies, I highlight the value of academic research. While it's far from perfect, it can be more reliable than practitioner studies for a number of reasons: Its goal is scientific inquiry, rather than advocacy of a pre-existing position or releasing findings to improve a company's image. It's conducted by those with expertise in conducting scientific research. Papers published in top scientific journals are peer-reviewed, which helpsimprove their accuracy. However, authors, journalists, and practitioners will sometimes cite research as if it bears the hallmark ...
Does only 2% of VC funding go to female founders?

Does only 2% of VC funding go to female founders?

A widely quoted statistic is that only 2% of VC funding goes to female founders. For example, this Forbes article highlights that "only 2% of all VC funding goes to women-led startups" and asks "Why is only 2% of VC funding going to female founders"? If true, this statistic is substantial underrepresentation and needs to be urgently addressed. However, it's problematic for several reasons. 1. The Statistic Ignores Diverse Teams The 2% statistic actually refers to companies founded solely by women. It ignores diverse companies founded by both men and women. This is strange, because ...
An unhealthy obsession with organisational health

An unhealthy obsession with organisational health

Two leading asset management firms drew my attention to the McKinsey Organizational Health Index as a potential tool to evaluate a company. A book, "Beyond Performance 2.0: A Proven Approach to Leading Large-Scale Change", written by two McKinsey partners, claimed that companies with high scores on this Index trounced their unhealthy peers along a range of performance measures. For example, their shareholder returns were three times as high. But as I wrote in an earlier post, rather than being more impressed by big numbers, we should be more sceptical. If it were really possible to ...