You and your A/B test: What are you really measuring?

Photo credit: Horwin.

Try this: Walk into a group of product testers and ask each of them what metrics they would use to determine which merchandising page—one showing discounts in percentages and the other showing marked down amounts—is the winner. I’m willing to bet my super cool desk plushies that 90 percent of them will be measuring the wrong thing for this A/B test..

But why would a lot of them get it wrong? Don’t be discouraged! One of the most difficult things about A/B testing is identifying whether a result is a causation or a mere coincidence. To better explain this, let’s take a look at an industry example.

Two Thai entrepreneurs Nam and Win want to design an app that will connect bakeries with nearby potential customers so that the latter can order pastries without having to wait in line. (Who doesn’t want an app that’s all about delicious confections?)

One of their first test ideas is to send users push notifications about daily specials and exclusive offers from nearby bakeries. They sent out the notifications on weekdays at 6:30 am (local time) thinking this is the perfect time to catch a majority of users before they start their daily commute. The campaign starts and they let it run for one week. On Saturday, they look at their data and find that they increased their sales by 25 percent. Awesome!

After celebrating a huge win, they realized that if one push notification increased their sales, a second one should have a similar effect. So, Nam and Win set up a second campaign where notifications are sent on weekdays at 4:30 pm, just before people start commuting back home. They run it for a week. Saturday comes around and they look at their data. Their sales didn’t increase. It stayed at the same level as when they just used a single push notification.

The wrong metric

Many would jump to the conclusion that Nam and Win annoyed the customers with too many notifications. Some might think that the customers only buy pastries once a day. The truth is Nam and Win have been measuring the success of their campaigns incorrectly. They assumed that a notification increases sales and that there was a causation between sales and notifications. (I’m not saying that notifications in general do not lead to sales, as there are many cases where sales can certainly be the goal for using notifications. But in this particular example, it should not be the metric.)

This brings us to the crux of the issue: the need to focus on the short-term goal, which prevents people from falling into the trap of assuming correlation always equals causation.

The right metric

Let’s now focus on the short-term goal for Nam and Win’s case. What was the message of their first notification? They wanted to inform customers about the daily specials and exclusive offers. Does this content really aim to sell someone an item? Is it specifically targeting the moment a customer is looking to make a purchase? Hopefully, you answered no because that’s correct!

Informing someone about the daily specials and unique items is really a fishing technique. This notification aims to change a non-visitor into a visitor—that’s it! The increase in sales was really attributed to the increase of visitors entering the “funnel” or user flow. Many testing philosophies fail to factor in the specific location in the user journey that their test is targeting. Nam and Win should have really looked to see if 0, 1, or 2 notifications gave them the highest increase to their daily visitor rate.

Still not convinced that this doesn’t aim to complete a sale? Let’s change up Nam and Win’s notifications. Let’s say their second notification was a bit more targeted and they only sent the second notification to customers who had opened the app during the day, browsed a specific bakery, but never made a transaction. This notification could say something like, “It’s still not too late to pick up that treat from [insert bakery’s name]!”

In this case, Nam and Win are no longer fishing. They are instead specifically targeting customers who have already shopped around in the last several hours but haven’t made a sale yet. These customers are at the point in their journey where they’re about to check out, and this notification aims to convince customers to actually purchase through the FOMO or “fear of missing out” tactic.

Take one last look at Nam and Win’s situation. They have sent a notification to customers attempting to increase their daily visit rate by targeting the very first step of the customer flow (opening the app) and have also set up a second notification to re-target customers who are nearly through the funnel and have browsed a specific item (the step just prior to checkout). By narrowing their focus and not getting trapped in the rationale that correlation equals causation, Nam and Win have a recipe for success.