A/B testing in B2C Startups

‍

Assuming you’ve determined the right balance of optimization and innovation from the above sections, we can now take a closer look at how to manage an optimization roadmap and pick the “right” experiments to run.

‍

Creating a Roadmap

‍

Like any good product team, you should begin with a roadmap. The roadmap should be organized in priority order with the priority determined by estimated impact and level of effort. For example, if you estimate that a certain set of tests can produce a large increase (double digit gain) in the metrics for a relatively small amount of effort (a few weeks or less of engineering and design support), then it’s likely a high priority experiment. I’ve also created a template for creating your own experimentation roadmap, which you’re welcome to make a copy of and run with it.

‍

The roadmap has two segments to it: The first segment allows for estimating the impact of various experiments so that you can rank them in priority order. The second segment is intended to capture the results from the experiment. It’s essential to maintain a history of all experiment results so the team can conduct post mortems in order to refine their experiment selection and design.

‍

Generally speaking, I recommend that optimization teams— such as a growth team—operate in 6-8 week sprints focused on improving one metric at a time. A common mistake I see is a small growth team trying to optimize multiple metrics in parallel. This lack of focus normally leads to subpar results. In contrast, significant results can be produced when the full weight of a growth team is poured into a single metric for at least a few months. The team will find that they improve their pattern recognition through focused effort, leading to better test results as time goes on. As an example, during my time at Quora, our growth team spent 16 months optimizing solely for sign up rate. During that time frame we increased the sign up rate from SEO traffic from 0.1% to north of 4%. Once we reached the bottom of the barrel on that particular metric, we moved onto the next metric and repeated the process. To encourage this type of focus, I broke the experimentation roadmap template into multiple tabs where each tab maps to a roadmap for a specific growth metric — e.g. churn vs. reactivation vs. signups and so on.

‍

Picking the “Right” Experiment

‍

Picking the right experiment to run is part art and science. By art I mean using judgement to craft a user experience worth testing. By science I’m referring to the practical constraints of testing new experiments on a relatively small population (i.e. sample size in statistics speak) when you’re still an early stage startup.

‍

I often see startups try to run A/B tests in the same way that large companies like Google and Facebook do. They create a list of A/B test ideas that require fairly limited level of effort and then they start shipping dozens of small change tests fairly quickly. A classic example would be making changes to the call-to-action on a landing page, such as on the homepage, and perhaps testing the location of the call-to-action as well. The problem with this sort of test is that a startup often has a much smaller sample size (because they have less traffic or users of the product), so running and resolving that A/B test at high statistical confidence takes much, much longer than running a similar test at a high traffic product like Facebook. The relationship between experiment thoughtfulness and sample size is captured in the below diagram.

‍

‍

Here’s how to interpret it: Companies with a large sample size (a lot of traffic) don’t have to be as thoughtful with experiment selection and design. The reason is that the large company can make relatively small changes to the product, set up an A/B test to measure the effect, and then resolve the experiment in a matter of days at high statistical confidence because they have a wealth of data to lean on. On the other hand, a small startup with very little traffic (small sample size) needs to be much more thoughtful about experiment selection and design because an A/B test on a small sample size that produces a small change relative to the control will take weeks or months to harvest enough data to reach a statistically significant conclusion. I’ll demonstrate this effect in the below table.

‍

Let’s imagine we have three different startups (A, B, and C — below). Each is going to run an A/B test on their homepage where the base conversion rate is 10%, the relative increase in conversion rate they are aiming for is 5%, leading to a new conversion rate of 10.5%. However, each startup has a different volume of daily traffic. Startup A receives 100 visits per day to the homepage, B receives 1,000 visits per day, and C receives 10,000 visits per day. Using the A/B testing calculator from AB Tastyto calculate the necessary test duration, we get the following results.

You can see from the data that the test duration declines significantly as a result of having more samples (i.e. traffic) in the test funnel. Now, let’s take a look at what happens when you tweak the magnitude of the relative experiment effect. In other words, when you run a test that produces a small, medium, or large change to the baseline conversion rate.

‍

‍

By increasing the magnitude of the relative experiment effect, the test duration declines precipitously. The key takeaway here is to aim for large changes. That seems like an obvious observation, yet I see many startups testing relatively minor changes to their product in the hopes it will produce a double digit increase in the target metric.

‍

Finally, let’s look at what happens if we manipulate the base conversion rate. By base conversion rate I’m referring to the starting conversion rate. For example, if you have 100 visitors/day to your homepage and 1 user signs up, and you’re running an A/B test on the homepage, then you have a base conversion rate of 1%. If instead you run an A/B test midway through the sign up flow where there are 10 visitors per day, and 1 visitor manages to sign up at the end of the flow, then you have a 10% base conversion rate. What you’ll notice in the below scenario is that test duration decreases as a result of having a higher base conversion rate. Practically speaking, that means you’re more likely to reach statistical significance quicker if you A/B test in the bottom half of a funnel versus the top half since the bottom half has a higher base conversion rate.

To recap, there are a few key lessons to take away from the above scenarios:

‍

1. Smaller startups can’t test like big companies because of sample size limitations. They simply don’t have as much traffic. If they try to test small changes to the product, which produces a small relative change in conversion rate on an already small sample size, then the test will take months or years to conclude. Startups don’t have the luxury of waiting around for insignificant results like that. On the contrary, startups need to produce step change increases in their rate of growth in order to achieve liftoff and set themselves up for another funding round.
2. Startups must test big changes to their product in order to manage sample size limitations. If a startup runs an A/B test for a significant product change that leads to a 30% worse conversion rate, they’ll find out in a matter of days and can quickly kill the experiment and limit the downside. If it turns out that the test produces a 30% increase in conversion rate, the company will also find out in a matter of days and can turn it live to 100% of users and experience a large increase in its rate of growth. When you think of it that way, the startup really has nothing to lose!
3. The bottom half of a funnel is often a better place to test than the top half of a funnel because obtaining statistical significance on a high baseline conversion rate is more likely than on a low baseline conversion rate.

‍

It’s essential that anyone working on an experimentation team or roadmap understands the above statistical concepts. If so, they are less likely to stack their roadmap with poorly chosen A/B tests that will take too long to run and produce results too small to change the trajectory of the company.

Creating Online Content Communities

‍

Flywheel Fundamentals

‍

Each year a batch of entrepreneurs set out to build the next great online community. Some attempt to build large horizontal platforms where users engage on topics ranging from immunotherapy to the Boston Celtics. Reddit would fit that description. Others seek to create vertical communities tailored to a particular subject and audience, such as Wheelwell for car enthusiasts. 

There are many reasons to be on the prowl for the next great online community, either as an investor or operator. A leading reason is that the winners tend to be massive. Another reason is that successful online communities seem to grow perpetually through organic growth, which is the holy grail in startup land.

The unstoppable momentum of organic growth, driven by users creating new content on the platform, is what is often referred to as the “flywheel”. In technical terms, a flywheel is a device that stores energy. The more it’s revved up, the more energy it stores and the longer it can spin unaided. It sounds like magic, but there’s a simple explanation for it—at least as simple as physics goes.

What a flywheel does is it converts kinetic energy into potential energy. Kinetic energy is the energy that an object possesses due to its movement. Potential energy is the energy stored by the object due to its position. Archery provides a basic example. When you pull the bowstring back, you can say that the arrow has potential energy. And when it is released the arrow has kinetic energy. 

Another key point about flywheels is that the bigger it is and the faster it spins, the more energy it stores and the longer it takes to slow down. Online communities that have established a “content flywheel” behave similarly. 

Let’s take Reddit as an example. The stockpile of registered users is Reddit’s version of potential energy. When those users create content and the content is discovered in Google, shared via social media, or distributed online through other means, then the “arrow has been shot” so-to-speak. In this analogy, the user-generated content is kinetic energy. When new content is created it fetches new traffic and users into the platform, increasing the size of the flywheel and accelerating its rotational energy. It becomes self-propagating. And once that kicks in, good luck stopping it.

To put the power of a content flywheel in perspective, Reddit recently claimed 430M monthly active users. It’s 15 years old and still spreading its wings. 

But how does one create such a platform driven by perpetual organic growth? Clearly, it can’t all be distilled down into a simple formula and bottled up in cans to be sold next to ketchup and mustard. It’s no commodity. There is no “secret sauce” that only Italian grandmothers and a few exceptional founders figure out. However, I do believe some of the ingredients are knowable and repeatable. This playbook will describe what those ingredients are, how they work, and what you can do about them in pursuit of building your own startup fueled by a flywheel. 

‍

The Ingredients

‍

I believe there are seven primary ingredients when it comes to building flywheels in software. Six of them are knowable and I’ll go into detail on each below. One ingredient is something only you, the founder, can figure out. It’s the “secret sauce” that makes your community stand out relative to the rest and is your unique innovation. 

Here’s the full set of seven ingredients: 

1. Basic flywheel design (flywheel 1.0): a high-level description of how your community acquires users, gets them to consume content, converts some users to creators of content, and how that new content leads to new traffic and users.
2. Consumption flywheel design (flywheel 2.0): a secondary flywheel that the product uses to accelerate the rate of content consumed within the community.  
3. Creation flywheel design (flywheel 3.0): a tertiary flywheel that the product uses to accelerate the rate of content created within the community. 
4. A “cold start” solution: a strategy for putting the initial momentum in the flywheel by identifying early adopters and coaxing them into becoming the first creators within the community.
5. Moderation and quality control: human-based and software-based solutions that maintain a bar of quality content creation and quality user interaction. 
6. Beachheads and vertical expansion: a strategy for establishing your initial user and content beachhead and a method for expanding into user adjacencies and new content verticals.
7. “Secret Sauce”: the unique “hook” that makes your community attractive, fun, and worthy of engaging with and that would entice users to ditch other communities in favor of yours. 

Let’s jump into each, how they work, and what you can do about them. 

‍

Designing the Content Flywheel

‍

Let’s assume at this point that you, the founder, have already decided on the type of content community you want to build. It could be for scientists, sports enthusiasts, Chief Information Officers, or be the world’s next horizontal platform to compete with Reddit, Youtube, and so on. It doesn’t matter which option you’ve selected. What matters is you’ve vowed to create a dent in the content universe.

You begin toiling away in your preferred design tool with product prototypes, first starting with low-resolution concepts. After a bit of user testing, you’ve identified a variety of UX snafus, shuffled around the deck chairs a bit, and arrive at a prototype that’s ready for development.

The designs turn into an alpha. You test it with more users. The alpha becomes a beta. You test it with more users. Finally, you’re ready to launch it. You turn on the TV and play the iconic scene from Field of Dreams where the spirit of Shoeless Joe Jackson whispers, “If you build it, he will come.” And like ghosts emerging from a cornfield, users show up and engage with each other like long lost friends. Hours and hours of lively conversations are created and your community is flush with chatter. 

Except, that’s not what happens. Conversations don’t spontaneously ignite and engagement is at a whisper. You’ve built it, but no one has come. 

This is where your journey to creating the flywheel begins.Kevin Costner’s journey began with designing a field, but yours begins with designing your flywheel and hand-picking your early adopters.

‍

Flywheel 1.0 - The Fundamentals

‍

Don’t get fancy. Start with 2,000-year-old technology and 500-year-old technology; paper and pencil. You don’t need modern software to design your first flywheel, so shut your laptop.

I believe there are four atomic units of a 1.0 flywheel:

1. Acquire: how users are acquired into the community (e.g. they sign up)
2. Consume: the mechanisms that drive content consumption (e.g. a newsfeed)
3. Create: the mechanisms that entice users to create content (e.g. social status)
4. Harvest: how new content created leads to more inbound growth (e.g. SEO)

Those elements represent the common building blocks of a content flywheel. 

It begins with a visitor signing up to use the product. After a user has signed up, the user will gain access to the stockpile of content that exists in the application, which they may start consuming. Note that the stockpile of content won’t exist at first. I’ll get to that in the section about solving the cold start problem. 

After consuming enough content, some users evolve into creators of content. The content the user creates leads to new traffic headed your way. A common example would be content indexed in a search engine or shared on social media, which fetches more traffic back to your community. Lastly, some of the newly harvested traffic will convert into newly acquired users that sign up to be a part of the community.

With a sketch similar to this, you can begin with a simple conceptual understanding of the content flywheel for your application. Assuming your product has launched and has at least a few hundred users, you can then measure the baseline conversion rates (CVR) at each step in the flywheel. 

In the above example, the conversion rate (CVR) for the initial signup is 1.5%. Of the users that sign up, 20% of them go on to consume content in the application and 5% of consumers go on to become creators of content. The new content that is created then leads to new traffic generated for the application.

In this example, I chose the metric of visits per piece of content per month. A practical example would be a question answered on Quora or a thread posted on Reddit. In this case, each question on Quora or thread on Reddit would receive an average of 2 visits per month. Finally, the new traffic generated from the content created by new users leads to brand new users signing up for the product at a rate of 0.2%, which is a fairly common conversion rate for long-tail traffic coming from SEO.

The conversion rate to signup from this traffic is typically lower than traffic that goes directly to an application’s homepage, such as someone opting to go directly to Reddit.com and sign up. Visitors navigating directly to an app’s homepage have relatively high intent, likely because someone told them about the product, which is why the conversion rate is highest for direct homepage traffic.

And just like that, you have your first content flywheel designed and instrumented with empirical metrics. But your homework isn’t done yet. You’ve only completed the first of three assignments. And this one was the easiest since most online communities have a nearly identical 1.0 flywheel. In fact, you can just copy this flywheel and you’re off to a good start.

‍

Continue reading Part 2: Accelerating Your Content Flywheel