You have said that every organization should embrace the practice of continuous experimentation. Why is this so critical, particularly right now?

The main reason is that experimentation is the engine of innovation. As a result, anyone who cares about innovation must also care about experimentation. Innovation is about novelty and value creation, but it is equally about uncertainty. There can be R&D uncertainty, production uncertainty, market uncertainty, customer experience uncertainty, to name a few. We typically deal with the uncertainty element by relying on our experience. But that can be quite limiting. The fact is, when you’re trying to create something new, in most cases you don’t have any prior experience with it.

Some might say, In that case, look to the data. But again, the same problem arises: If something is going to be novel, by definition, there is not much data around it. That leaves us with the notion of experimentation, which allows us to test what works and what doesn’t.

Like everything else, experimentation is changing. There are companies out there running thousands — if not tens of thousands — of experiments as we speak. They are doing this online, they’re in brick-and-mortar environments, and across B2B and B2C. The results are affecting everything these companies do, and this practice is incredibly powerful. If you don’t understand how to experiment and you aren’t doing it at scale, you are at a competitive disadvantage.

You have said that in the realm of experimentation, success and failure co-exist in a paradoxical balance. Please explain.

Failure is very important to innovation. If you already know that something works, it’s not really an experiment. The companies in online spaces that I’ve studied tell me that they fail eight to nine out of 10 times. It’s just a normal by-product of innovation for them.

If something is novel, by definition, there is not much data around it.

I do want to make a distinction between failures and mistakes. Mistakes do not add any value, and they should definitely be minimized. Failure is different because it usually gives you an answer to a question. There is a learning objective involved, while with mistakes, there is no learning objective.

Take a company like Amazon, which has numerous distribution centres and warehouses. If they set out to build yet another one, there is really no innovation or learning objective involved. It’s just operational execution. If the project goes wrong, it means they screwed it up. Clearly that’s not a good thing, and it is very different from innovation, where the initiative is all about learning — and fast.

Incrementalism doesn’t get much positive press, but you believe there is a place for it. Tell us about high velocity incrementalism.

Innovation comes in many different flavours. People often think of it in terms of breakthroughs or disruption. But it turns out that most innovation in the world today is incremental — and that is not a bad thing. It is actually very powerful right now in a digital context, because when you can scale something instantly to hundreds of millions of people, even a small change like a two or three per cent improvement can have a massive impact on revenue.

Breakthroughs and disruptions don’t happen very often, so focusing on incrementalism can be a good thing — as long as you do it fast. This is where the high velocity aspect comes into play. We need to be iterating — responding to feedback and pivoting — very quickly. As indicated, a significant performance change can be the result of many high velocity, small changes. That’s exactly what we’re seeing with a lot of online businesses.

Talk a bit about how human foibles like hubris and bias can get in the way of innovation.

This goes back to how we deal with uncertainty as human beings. We really don’t like it; so we often rely on experience and expert opinions. Many senior executives feel like they get paid to make tough decisions, so they want to make them quickly. But when it comes to innovation, as indicated, we are wrong most of the time. Even experts are wrong most of the time when it comes to predicting customer behaviour.

Behavioural experts like Daniel Kahneman have shown that there are lots of reasons why we make these kinds of mistakes. We mistakenly attribute cause and effect to what are just random events; we weigh losses much more heavily than gains; and we tend to happily accept good results that confirm our biases, but to challenge bad results that go against our assumptions. Exactly 400 years ago, Francis Bacon wrote Novum Organum, a philosophical work about exactly this — how humans are not great at processing information because we get fooled by our own hubris. Experiments are a very powerful mechanism to correct these human weaknesses — and keep us honest.

You have extensively studied the largest accommodation platform in the world, Booking.com. Describe how it embraces the experimental mindset.

One of the key attributes of an experimental organization is that it democratizes experiments. Booking.com runs a massive number of experiments — in the tens of thousands each year. They use these to optimize the landing page that users see when they come to the website to find a place to stay. Prior to the global pandemic, they had more than 1.5 million bookings per day on their site.

The company has democratized experimentation in the sense that any employee can launch an experiment on millions of customers without getting permission from management. Something like 75 per cent of its technology and products actively use the company’s experimentation platform. So they are — or they were, and will be again — experimenting continuously.

They also have interesting checks and balances in place to make sure people don’t launch bad or unethical experiments. Employees who want to try something have to be totally transparent about their experiments. Before they launch it, they have to show it to everyone on the platform, and people can provide feedback right away. That’s a very democratic way to check up on each other.

Also, anyone in the company has the power to stop an experiment. Of course, no one would do that unless they felt like something really bad could happen. This is a very interesting company in terms of understanding the culture you need to create to make experimentation work on a large scale.

The goal is for people to run experiments at large scale, without management permission.

You believe that every organization should invest in its own large-scale experimentation system. What do these look like?

There are different elements. Companies like Booking.com, Amazon or Facebook have their own internal platforms for experimentation, because when they got started on this there were no third-party tools around. But today you can get tools that help you set this up. The three big pillars involved here are process, culture and management. I have identified seven components — I call them the Seven S’s: scale, scope and speed (which I group under process); shared values and skills (culture); and standards and support (management).

You believe that along with the great power of experimentation comes great responsibility. Can you please touch on that?

I think a big piece that needs to be discussed a lot more is the ethical side of experimentation. Whenever you begin to contemplate a new experiment, you must think carefully about whether users will consider it to be ethical. The answer isn’t always clear-cut, so you need to examine that question if you don’t want to spark a backlash. As we have seen, Facebook has failed to do this on more than one occasion.

In medicine they say, do no harm, and I believe that is always a good guiding principle. Sometimes the stakes are not high — such as when you want to change a colour or a button on a landing page. Nonetheless, ethical considerations need to be part of your culture.

It’s a tricky balance. You want all your people to run experiments at large scale, hopefully without management permission. But that, of course puts the burden on employees to make judgments. Some companies create a peer-review committee that goes through an approval process — similar to what we have in hospitals and universities. But if you want to go for scale, a review board will slow things down and you will not be able to run tens of thousands of experiments. You need to create systems that delineate these kinds of things. There are certain kinds of experiments that have to be really thoughtful and the ethics have to be thought through, versus other kinds of experiments where you can put your foot on the pedal very quickly.

Booking.com, for example, has ethics training as part of its onboarding process. And LinkedIn has ethical guidelines that it has created for their people, making it clear that certain kinds of experiments are not okay. That’s what I mean by great power coming with great responsibility.

Sadly, Booking.com—one of the companies that does all of this so well—is being hit hardest by the global pandemic. What is it going to take for them to rally?

Sadly, travel is sort of done for some time, so any company in that industry has been severely affected by this. It’s important to remember that these are unusual times and to keep morale up. If anything, I believe experimentation and innovation will get us out of this crisis. The capability that got Booking.com to where it wanted to be will also be an important capability to get through this crisis.

I have been writing about testing for experimentation for more than 25 years now, and I’ve never seen so many articles on testing as I have in the past few weeks. If any good comes from this, perhaps it will be a heightened awareness that smart testing is really, really important, whether you’re dealing with a health crisis, a business crisis — or just day-today operations.

It’s too bad that we had to discover the great value of testing through a crisis, because the best time to develop a great testing capacity is when you’re not in a crisis. In the U.S., we have been paying the price for that.

There is no doubt in my mind that having a smart testing strategy is at the heart of not only solving the public health side of this, but also for dealing with the economic recovery. Without smart testing, we won’t have any good data, and when you don’t have good data, it is extremely difficult to make good decisions. And that applies to every organization out there.

---

This article originally appeared in the Fall 2020 issue of Rotman Management magazine.