The art of moving off of an A/B Test system
By Marty Bacik
And how to not make it a horrible experience for your users
At Calm, as I assume is similar with many consumer tech companies, experimentation is a way of life. After all, you can speculate as to what the customer may want or like, but nothing beats having hard data to validate the assumptions we've made. We seldom add something to the app without A/B testing it first.
At the time of writing, I’m in the process of moving Calm away from not only our first in-house experimentation system, but subsequently away from our first external experimentation system. I don’t claim to be a SME in many things, but when it comes to deprecating old A/B test systems, I’m somewhat of a pro.
There is a reason I call this an art - experimentation systems are unique beasts. For one, the business logic controlled by experiments is likely spread widely throughout the code. This is especially true if you have accumulated some tech debt related to closed experiments which still remains in the code (no judgment, I’ve definitely been there!). Because of how many proverbial tentacles these systems can have, there is a reasonable amount of risk involved in breaking something fairly significant or noticeable for your users when migrating systems.
There are a multitude of ways you can approach experimentation, ranging from in house to external vendors. As a result, there will likely be a point in your career (or many) where you may be tasked with helping to move from one experimentation system to another. If you’ve been tasked with this, fear not! I’ve laid out 4 steps below from my experience to make this as painless and seamless as possible.
1. Zoom Out
As engineers (myself included) I think we like to dive right into the code and get our hands dirty as quickly as possible. Now is the time to fight that instinct. I promise that you will thank yourself if you spend some time understanding the big picture of your systems and everything that is touched by your current experimentation system first. For me, I made an Excel sheet of our 300+ experiments (boring), wrote down the state (closed, running), and what variant won if closed - pictured below.
Not only did this planning phase make the removal of our old system almost seamless for our users, but it gave me data points and a history for when someone from the business asked why I hardcoded a feature a certain way.
2. Start With Clients
I recently purchased a chainsaw and spent the summer cutting down a few trees, so please forgive my Western Pennsylvania centered metaphor in advance. You can think of the server/API as the base of the tree when it comes to experimentation systems. The clients (iOS app, Android app, web app, etc) are the branches. Your experimentation system is then the roots of the tree. Your goal is to cut down the tree at the base, but if you’re cutting down a tree, it makes sense to remove as many branches as you can first. Otherwise, they may fall onto a house, powerlines (I may or may not know this from experience), or something else. Chaos!
This same methodology applies to removing a legacy A/B test system. You’ll want to remove client code first, especially any calls to your servers experimentation endpoints. You can remove the server side endpoints first, and this could be fine. But it also could have drastic consequences to the user. If your clients try to call GET /experiments, but you’ve already removed that, yikes! Trim your branches first or else they might fall on a powerline! The last thing you want is having an experiment control something like your homepage, and your app changing for all users as a result of removing the experiments.
With any clients you have on app stores, there is the additional complexity of releases and versions. You’ll want to give yourself some lead time for people to upgrade their app versions. And, ideally, hard block the old versions of the app to prevent unforeseen issues. Last time I removed an A/B test system, I removed the iOS and Android app code in January, but didn’t alter the server side code until March. You don’t have to wait this long, but generally giving your clients some lead time will leave you in better shape.
3. Move the Server to a Snapshot State
If you’ve made it this far beyond my weird metaphors, you know by now that the focus is on retaining as much structurally as possible when removing the existing system. After you’ve moved the clients away from the legacy experimentation system, you’ll effectively want to take a snapshot of experimentation functionality today, and hard code or retain that definition into the foreseeable future.
For example, if your /experiments endpoint is returning a JSON schema of certain test names and variants, grab that response, make a note in the code for anyone looking at it in the future, and return those tests as a JSON constant from this endpoint. Here is a snippet of the static response our server returns for our double legacy in house experimentation system:
// The GET /tests endpoint was deprecated in all clients, yet
// there are still requests from older clients to this endpoint.
// Grabbing a snapshot of the response on April 17, 2023 and returning
// this as the endpoint response.
const ABTestSnapshot: Record<string, ABTestEnrollmentSanitized | undefined> = {
homepage_free_trial_2018_04_02: {
variant: 'free_trial',
winner_picked_at: '2018-04-26 00:58:41.963+00',
is_enrolled: false,
},
EoS_reminder_time_of_day_ios: {
variant: 'enabled',
winner_picked_at: '2020-05-13 23:21:01.996+00',
is_enrolled: false,
},
homepage_h1_text_2017_08_14: {
variant: 'welcome_to_calm',
winner_picked_at: '2017-11-22 02:02:36.694+00',
is_enrolled: false,
}
}This will retain the functionality for legacy clients while removing external calls to the experimentation system. If there is specific business logic around experiments, use your knowledge of the overall system from step 1 to hardcode these business rules instead of using the variants (see, aren’t you glad you did the leg work up front to have this knowledge!).
4. [BONUS] Optimize
There are few times in our daily work as engineers where we get to take an overall view of the system outside of our tickets and get a feel for what can be cleaned up. Take this opportunity to optimize (or propose an optimization) for areas you find that can be improved. Sure, the initial scope may have just been to remove an external call to an experimentation system from a function, but maybe you determine the function itself has been sitting unused for a while. To go back to the tree reference, cut it down! Once your work is done, you can have a much cleaner codebase (a forest with nicer trails) than when you started.
I hope this post gives you more confidence to move forward with moving from your own experimentation system, or at least has given you more appreciation for the unique nature of experimentation systems as a whole. As always, engineers have an obligation to the products and users we serve in addition to the technology. You could take a technologically sound approach to ripping out an existing A/B test system in a matter of hours and call it a day. But, in this instance at least, taking the long road is the best route to serve the customers we ultimately come to work for every day.