AB testing or Pre-post

When to run AB test and When to go ahead with Pre-post analysis

Teams do a fair share of debate on when should they do AB testing, and when should they rely on pre-post analysis. These are methods to quantify the impact of a product change or marketing push.

No one answer fits all the use cases. This post lays out a mental-model to think through it? It will list out a set of questions that will help you choose the best method.

What is the goal?

Treatment must follow the diagnosis. AB Test or pre-post are two different treatment plans. You must delve deeper into the business question for the diagnosis.

Yes, you want to quantify the impact of a product push. Considering impact quantification is not a binary question, you need to understand the side of the spectrum you need to be.

As an analogy, think of measuring tools. The measuring tools needed in a home kitchen, a restaurant kitchen, a college laboratory, and a nuclear reactor are not the same. All are measuring tools. You may use a science lab measuring tool in your home kitchen. But, it will make you Sheldon.

Doing AB testing for all impact quantification questions is not smart. You don’t need to throw time and resources to get the accuracy you don’t need. It slows down innovation as well.

A set of seven Questions

Let me start with a question. It is often the reason behind the debate on the pre-post vs AB test.

Are you having the debate before the product push or after the product push?

Well done, if you have not pushed the product yet. I’m sure you are going to make a sound decision because you are having the debate before the launch.

I’m not surprised if you have already done the product push? Are you getting a hard time from another BU? Do they believe the product push has a negative impact on their north star metric?

Believe me, AB test is not the solution here. Of course, it will help you buy time while you work on solving the real challenge. This is a sign of dysfunction within the organization. It could happen even in a well-run organization. But, mature organizations, most of the time, are able to solve it with pre-post exercise and healthy debate.

Now here are the set of questions. Make the team debate on it, and you will know the approach that makes sense for your use case.

1. How accurate we need to be on impact quantification? Can we live with lower accuracy here? Remember, there are factors that don’t get accounted for even in the AB test read-outs.
2. Are there other metrics that will give you more confidence in the pre-post result? The answers, more often than not, is yes. A lead and correlated variables will give you more confidence in pre-post results. It is good to do even for an AB test.
3. How does the product-push fit into your product strategy? Is it part of an uber product strategy that the team already has the plan to execute on. Is it possible that the executives may recommend you to go ahead with it unless the number is super negative?
4. Is the product push one simple change, or you plan to keep playing with five other components to get to the final experience? Knowledge is power, but do you think quantifying the impact of a component of the experience is worth the effort? The cost is delaying the launch. Remember, all three components must go together.
5. Do you have a simple way to define synthetic control that can give you more confidence in pre-post? More details to follow.
6. Do you have results from other hypothesis validation methodologies? Surveys, qualitative methods ( focus groups and interviews). Consistent results from a couple of independent, low fidelity methods may give you the confidence to go with less involved quantitative methods.

A graphical mental model that connects the AB test and Pre-post

Here is how you can think about an AB test and pre-post. It will help you be more confident about the methods. The confidence, in turn, would translate into swift decision making when you have to choose one method over another.

1. The benefit of the product push or marketing campaign at time = t is ( D- d). For a perfectly randomized AB test, ‘d’ will be zero. However, that is never going to be the case. Let us just say that d → 0.
2. Please notice how the value of control also changes post push. It could be because of a factor outside your test — seasonality, industry-level changes. It implies that C(t=t) is not equal to C(t=0).
3. Now, let’s say, you don’t have control and want to understand the impact of your product push. What you want to calculate is T(t= t) and T^ (t=t). T^ is the dotted line. By the way, for an AB test when you do D-d, in fact, you are calculating the distance between T and T^. ‘d’ is the distance between C and T^.
4. Unfortunately, you don’t have T^ as you don’t have control. The best you can do now is, try to get the best estimate of T^. You could say, T^ at time=t is equal to T^ at time = 0. Or, you could say T^ is average of T in the pre-period for the X period. This is what all pre-post analysis is doing. So, in a simplified pre-post analysis you are making an assumption that T^ is not going to change in the post period. If it holds your work is as good as any AB test.
5. In more advanced pre-post estimation, you may like to estimate the value of the difference between T^ (t=t) and T (t=0). You will find out a comparable population for T (say T*). You will look at the difference between T* (t=t) and T *(t=0) and declare it as the difference between T^ (t=t) and T (t=0). Finding a comparable population is an art. Advanced statistical methodologies help, but domain knowledge helps even more. Some call it synthetic control.