The fascinating thing about intuition is that a fair percentage of the time it’s fabulously, gloriously, achingly wrong
— John Quarto-vonTivadar, FutureNow

It is now becoming a mainstream to talk about culture of experimentation (Always Be Testing). Many companies after years of driving this fundamental change into their cultures are starting to flood us with tons of fascinating information about their tremendous successes in this endeavors. Some success stories are worth mentioning like Intuit, Amazon, Microsoft, etc…

Here is a small excerpt from this amazing work published by Microsoft members of ExP team:

The statistics we share about the percentage of ideas that pass all the internal evaluations, get implemented, and fail to improve the metrics they were designed to improve are humbling.

It is humbling to see how bad experts are at estimating the value of features (us included). Every feature built by a software team is built because someone believes it will have value, yet many of the benefits fail to materialize. Avinash Kaushik, author of Web Analytics: An Hour a Day, wrote in his Experimentation and Testing primer (Kaushik, 2006) that “80% of the time you/we are wrong about what a customer wants.” In Do It Wrong Quickly (Moran, 2007 p. 240), the author writes that Netflix considers 90% of what they try to be wrong. Regis Hadiaris from Quicken Loans wrote that “in the five years I’ve been running tests, I’m only about as correct in guessing the results as a major league baseball player is in hitting the ball. That’s right – I’ve been doing this for 5 years, and I can only “guess” the outcome of a test about 33% of the time!” (Moran, 2008).

Evaluating well-designed and executed experiments that were designed to improve a key metric, only about one-third (1/3) were successful at improving the key metric!

It is mind boggling to see all the evidence that shows how we, as human beings, are constantly wrong with our assumptions. We are driven by emotions and not by data (Damasio Desscrats’ Error). So we have to isolate the decision making from pure emotions and rely only on the hard evidence – the evidence backed up by a lot of real data.

It is not always simple to get the required data to make a decision and it is not always straightforward to see how to collect that data. I’m not going to sugar coat it – collecting and measuring the RIGHT data is hard. The best way to choose the right data is through the experimentation.

Those who figured out the power of experimentation threaten to shudder the foundation of modern innovation, business agility and product development. With the speed of innovation and change, if you do not embrace the change, you will face extinction. You’ll become irrelevant, outperformed and out experimented by your much smaller but agile industry peers and competitors (Netflix vs. Blockbuster).

The companies that put experiments as a primary driver of their business and product decisions are, inherently, save a lot of time and effort from being wasted on the ideas that don’t work.

ImageHere is a very interesting data from the CHAOS Manifesto (Standish Group 2013), that only 20% of the features are used often and another 30% are sometime used.

Interestingly enough the usage of the features is not staying constant. It evolves over time along with the product since the customers’ interests and the market requirements change. “The Law of Diminishing Returns” from the CHAOS Manifesto 2015 states: “The value of features and functions change over the age of the application. As the application ages, the value of the features and functions used decreases”.

This tells us that the companies that concentrate first on the ideas that are bringing the most “bang for the buck” are going to get the most out of their efforts. As well as the companies that are constantly reevaluating the existing assumptions are the fastest ones to shed the old code and features that outlived their usefulness, reducing architecture complexity and increasing the current velocity.

So the only way to find what to do and when to do it is to constantly question your assumptions and validate them through myriads of experiments. Make it safe in your organization to admit that you and everyone around you may be wrong and make it super simple to run experiments. Build and architect your products for experimentation.

I’d like to summarize my thoughts with a couple of great lessons presented (video, slides) by Jeff Patton in his amazing keynote  “Won’t Be Fooled Again. How organizations have evolved to value learning over self-deception”:

  • This is hard, we’re usually wrong. Plan to learn
  • Engineer first for experimentation, then focus on scalability and performance
  • All we need to do is build-measure-learn…

To experiment effectively, it takes time, money, and whole organization participation. All this, and and a culture that makes it safe to learn.

Would like to hear how your organizations make it safe to fail and experiment.

P.S. Stay tuned for the follow up article about the “Experimentation with Culture”. As a home work think about this: if we’re mostly wrong with out assumptions, so why do we think that we know what company culture is right and why do leaders assume they know how to build the right culture?