Have you ever worked with colleagues who rely too much on gut feel or intuition to make important decisions?
According to Stefan Thomke, Professor of Business Administration at Harvard Business School, intuition can frequently lead people down the wrong path. In his new book, “Experimentation Works: The Surprising Power of Business Experiments,” Thomke argues that relying on past experiences, intuition, and even big data, won’t lead to innovation with impact. Instead, he says, delivering positive outcomes depends on the organization’s ability to run effective experiments and create a culture of experimentation.
In an interview with InnoLead, Thomke discussed how companies can become “experimentation organizations,” highlighting data from his research on companies like Booking.com, Microsoft, and Amazon.
Why is experimentation important to innovation?
As innovators, we often don’t know what doesn’t work, and the ability to predict customer behavior is extremely difficult. In fact, in the online space, some of the best companies get it wrong about nine out of 10 times. The big challenge is uncertainty, which makes innovation really hard.
There are different kinds of uncertainties that we face every single day in a business when we’re trying to innovate. One type of uncertainty is R&D uncertainty — [will] a product or service and technology…work as intended? But then we’re also faced with production scale-up uncertainty, which means just because we can get it to work in R&D, it may not work when we try to scale it up. The third [uncertainty] is customer experience uncertainty. Just because we can make it doesn’t mean that the customers actually want it, and if they want it, are they willing to pay for it? The fourth one is…when you’re moving into a new business space. What exactly is the ROI? Is the opportunity big enough to justify the resource investment?
How do we deal with that uncertainty in innovation? Well, quite often, we rely on experience. [However,] experience limits us in some ways, and maybe even takes us down the wrong path. … We now live in a world of big data and analytics. Big data and analytics are great, but it has a number of challenges. When you’re creating something that’s very novel, by definition, there’s less data. If there’s a lot of data around it, that means someone has already done it before. We also have problems with context. … What works in one setting doesn’t work in another setting. And then, finally, the big problem is that when we look at things in the world around us, and we look at our own experiences, we often see correlations between things. … For example, palm size correlates very highly with life expectancy. … There’s a strong correlation, but of course, the underlying variable here is gender. Women on average, have smaller palms and tend to live longer.
So, what do we do? That brings me to experiments. Experiments are there to test things, it’s our moment of truth. … The market will give us the feedback that we need…or customers will give us the feedback. And so experiments are the engine of innovation. Building a capability of experimentation needs to be an important part of what innovators do today. And when I say experimentation, I don’t mean just trial and error. I mean, having a disciplined approach to running experiments.
What does it take to build an ‘experimentation organization’?
An experimentation organization is an organization where experimentation is a little bit like breathing. It’s something that happens constantly; it’s built into the organization. Nobody questions whether they should run [an experiment] or not. … You have an [experimental] culture, not just the tools [to experiment]. … I’ve identified several elements that are necessary for having an experimentation culture.
- Cultivate curiosity. Often when you experiment, it’s about surprises. … If you run an experiment or a test, and there’s no surprise, there’s no learning because you got exactly what you wanted. So, you need to have an organization and people that are curious… If you have an organization where surprises are not valued, where you need to plan everything in advance, and where surprises are considered to be distractions or even something negative, it’s going to be very difficult to build [the experimentation] mindset and culture. Curiosity needs to prevail.
- Insist that data trumps opinions. One of the big issues that we have in innovation is that a lot of decisions are driven by either opinion or by intuition. … Humans by nature are driven by either prejudice or by cognitive dissonance. The danger is in an organization, if we are driven by biases or by opinions, we tend to make mistakes. There’s an interesting moniker that is used in the experimentation community called “hippo,” the highest-paid person’s opinion. … When you go very high up in an organization, people have very strong opinions and they also have power. They tend to force their opinions on people. If you want to create an experimentation organization, you really need to create an environment in which the decisions actually follow the data — where the data trumps opinions.
But that doesn’t mean that you have to blindly follow experiments. There is a great example from the Wall Street Journal about Netflix. … With the show “Grace & Frankie,” the Netflix series…they ran a test promotion that had an image of only Lily Tomlin [but not Jane Fonda]. … They found that when they run the image of Lily Tomlin alone, it results in more clicks by potential viewers than any promotions that have both Tomlin and Fonda. So, now they have a dilemma. … The concern was that if they exclude Jane Fonda, they would alienate the actress and even possibly violate the contract. They had a lot of debates inside Netflix — empirical evidence against strategic considerations. They decided to use an image [of both of the actresses], even though the data told them that it’s not as good from a customer engagement perspective. But what happened here is the experimental evidence actually added clarity to the decision, so at least they couldn’t pretend that they’re running this image of both actresses because customers would be more responsive.
The data needs to play an important role in terms of how you make decisions, but it doesn’t mean that you have to always blindly follow the experiment when other considerations are in play.
- Democratizing experimentation. Democratizing experimentation is kind of interesting. [For example], Booking.com is one of the most visited travel accommodations platforms in the world, and they run a huge number of experiments. … They run [an estimated] 25,000 [experiments a year]. But what’s even more interesting is that at Booking.com, any employee can run a live experiment on millions of customers without management permission. About 75 percent of its core technology and product staff, which is around 1,800 [employees]…actively use the company’s experimentation platform. They’re experimenting all day long. … What’s also interesting from a cultural perspective is not only can anybody launch an experiment, but anyone in the company can also stop an experiment. In order to do that, you have to be completely transparent. … And that’s what I mean by democratizing.
But of course, when you start doing that, you’ll obviously run into issues. The concern often is what if an employee actually launches something that brings down the website, or they launch an experiment that’s offensive? Which brings me to the fourth one…
- Be ethically sensitive. Ethics is an important part of being an experimenter, because the immediate reaction [from customers] is “What do you mean they’re experimenting on me?” It’s often surprising to people that they’re being experimented on. … We need to be careful about that. And sometimes, when companies are not careful, it can really backfire. There’s a very famous experiment that Facebook ran in 2012…[based on] emotional contagion. … [If you] walk into a room and you’re super optimistic, your emotional state rubs off on the people who are in the room. When you’re happy, they’re happier, and the opposite is true as well. Facebook wanted to know whether these emotional states were also contagious on its platform, without any physical proximity. … They wanted to know if you read more positive words, are you more likely to send out more positive words, and same for negative words. So, they did this as part of a study and then they published it. You can imagine the reaction — it really blew up. People got really upset about being manipulated. And the actual journal that published the article had to issue an apology. It triggered a huge discussion on ethical standards.
You have to ask yourself as an organization, what is ethical here? And you have to include that in the training of new employees. You have to have those discussions, because it’s never clear-cut. …Companies like Booking.com have very passionate discussions about [experiments] that are close to the line, and people will come out on different sides. You have to also build that into your culture — having open conversations about what’s ethical and what’s not ethical.
- Embrace a different leadership model. If all decisions are subject to experiments, what exactly is the role of a senior leader? We always educate senior leaders to be decision-makers, and if suddenly we leave decision-making to experiments, you have to rethink, what is the role of senior leaders? The first role is to set what I call a “grand challenge.” You can’t just have people in an organization experimenting willy-nilly, because they may be going after very different things. It’s important to give them a sense of where the top of the mountain is — what exactly is the grand challenge? And then allow them to take the grand challenge and break it down into testable hypotheses and key performance metrics, and make it part of an overall experimentation program. It could be something simple, like, “Our grand challenge is to create the best possible experience for our customers.” And now we can break it down in terms of what that means. [What are the] different components to having a great experience when they interact with our company? Then, let’s work on sub goals, write down different hypotheses and the relevant metrics, and make that part of this program.
The second role is [that senior leaders] have to put in place the systems, the resources, and the organizational divides to allow this to happen. Without systems and resources, this is really hard. To do this, you need infrastructure, tools, data pipelines, and so on. Third-party tools are now available that make things a lot easier, so you can get started pretty quickly. You have to think about how centralized should this be? Can we actually push it into the different business units? Can we decentralize it to people who really understand this? There are a lot of questions that senior leaders have to think about, but without the systems, the resources, and the organizational design, it’s unlikely that they’re going to scale things.
The third [role] is senior leaders have to be role models. They have to live by the same rules as everyone else, and they have to subject their own ideas to [experimentation]. If you have a big ego, it’s not going to work. … When I did some of the interviews at Booking.com, there was a new CEO who came in and wanted to make a [change] in the logo. The CEO said, “This is what we’re going to do with the logo,” and the employee said, “Okay, we’re going to test it and see if you’re right.” Even the bosses have to be challenged. The bosses have to demonstrate intellectual humility, and they also have to be unafraid to admit that they don’t know.
How do you combat the fear of failure?
There has to be a recognition, especially in innovation, that failure is just part of the game. [My advice to senior leaders is] you have to make a distinction between failures and mistakes. … A mistake, for me, is something that has no value-add. There’s no learning going on. For example, if I’m at Amazon and I’m building another warehouse, and I’ve built so many of those, it’s really just operational execution. If something goes wrong, it’s a sign of poor execution, it’s not a sign of tremendous learning. That’s different from failure — failure has a learning objective.
When you think about the numbers that I gave earlier — that in the online world nine out of 10 things fail — failure is super normal. So, when I go into organizations that do this at large scale, there’s not much discussion about failure, because it’s just so normal. … In fact, getting something back that works is abnormal. If you’re only running 10 experiments a year, and you only get a hit rate of 10 percent… then failure becomes a big deal. But if you’re running 1,000 experiments a year and 100 of them work, then you don’t worry too much about the other 900. …
We have to send people the right signals. Of course, we don’t want them to make mistakes because they don’t add value. But we want to draw a clear distinction between mistakes and failures. We want to encourage them to make failures, especially if they’re early on. … And we need to have a model where the cost of the failure is very low, so we can pivot and move on to the next one.
Thomke’s book, “Experimentation Works,” comes out on February 18, 2020.
