How to Turbocharge an experimentation program

A step-by-step guide on how to ramp up an AB testing program

Years of debate and execution have taught us a lesson or two on how to ramp up an experimentation program. This post shares those learnings and frameworks.

First, my credentials as it relates to experimentation. At a public payment company, my team was the power user of the experimentation platform, marketing platform, and engagement platform. The experimentation need of our team resulted in many new feature request for the platform. I got a good view of the development of a world-class platform.

In my last role, I lead the experimentation program for a real estate media listing company. We quadrupled our experimentation velocity in one year while making the decision-making process more robust.

It is going to be a series of posts. The first one will cover the ‘what’. The subsequent posts will cover how can we execute it.

What is an experimentation program?

First, let’s define the experimentation program. Experimentation is a way to validate and quantify a hypothesis. The charter of the experimentation program is to make the hypothesis validation process cheap and efficient. An experimentation program enables an organization

1. To convert ideas into a testable hypothesis. Help set up experiments with valid and robust test design
2. Measure the impact of experiments. Champion processes and methodologies that drive high-quality decision making
3. Build and enhance infrastructure, tools, and platforms for quick and cheap hypothesis validation

Keeping the three pillars of the program in mind, let’s get into the steps.

I have a benefit of the hindsight. So, rather than following what we did, I am putting down steps that seem logical now. Depending on the experimentation maturity, culture, and organization structure, your challenges may be different. But, I am sure the framework would still be applicable.

Problem Discovery: What are you really trying to solve

“If I had an hour to solve a problem, I’d spend 55 minutes thinking about the problem and five minutes thinking about solutions.” — Albert Einstein

A problem well stated is a problem half solved. The design thinking approach is the way to go. Please start with a conversation with all X-functional stakeholders and the participants to list out the pain points.

Step 1: Talk to executives to understand what they want to get out of it. Most likely, it would look like below

Step 2: Talk 1–1 with each X-functional team to brainstorm a solution for all the questions. I’m sure it is similar to what we have below.

Step 3: Rationalize the list of solution

You can group the feedback into three groups like below. a) Program management, process improvement and automation b) Enhancing platform, technology and data stack c) New Methodologies and Best practices

The second dimension that you have to slice all the feedback is by the stages of running the experiment. There are three stages of an experiment a) pre-test b) Test duration c) Post-test

Every minute of pre-test work may indeed save you 10 minutes of execution. So, please don’t push the items that you must take care pre-test

here is how the list of solution would look like

There are items that fall under multiple ‘improvement categories’. Some items you need to address not only before a test launch but also at the time of the test. For some items, you may not agree on the groups, but that’s okay. We’ll dive deeper into it as we get to the next post.

Conclusion

In this post, we laid out a framework to find out what we need to do to ramp up an experimentation program. Next, we’ll discuss how can we execute the solution that we have identified.