Introduction
It might surprise you to hear that effective growth isn’t all about the numbers. Focusing exclusively on data and optimization can subtly overtake innovation and leave you with a lackluster product. Besides A/B testing, long-term growth requires outsized bets and an organization designed for innovation.
This chapter will explore product testing techniques, designing an organization for growth, and creating content communities to help you gain traction with users.
What You'll Get
00 guides
00 tools
This chapter will cover
Assuming you’ve determined the right balance of optimization and innovation from the above sections, we can now take a closer look at how to manage an optimization roadmap and pick the “right” experiments to run.
Like any good product team, you should begin with a roadmap. The roadmap should be organized in priority order with the priority determined by estimated impact and level of effort. For example, if you estimate that a certain set of tests can produce a large increase (double digit gain) in the metrics for a relatively small amount of effort (a few weeks or less of engineering and design support), then it’s likely a high priority experiment. I’ve also created a template for creating your own experimentation roadmap, which you’re welcome to make a copy of and run with it.
The roadmap has two segments to it: The first segment allows for estimating the impact of various experiments so that you can rank them in priority order. The second segment is intended to capture the results from the experiment. It’s essential to maintain a history of all experiment results so the team can conduct post mortems in order to refine their experiment selection and design.
Generally speaking, I recommend that optimization teams— such as a growth team—operate in 6-8 week sprints focused on improving one metric at a time. A common mistake I see is a small growth team trying to optimize multiple metrics in parallel. This lack of focus normally leads to subpar results. In contrast, significant results can be produced when the full weight of a growth team is poured into a single metric for at least a few months. The team will find that they improve their pattern recognition through focused effort, leading to better test results as time goes on. As an example, during my time at Quora, our growth team spent 16 months optimizing solely for sign up rate. During that time frame we increased the sign up rate from SEO traffic from 0.1% to north of 4%. Once we reached the bottom of the barrel on that particular metric, we moved onto the next metric and repeated the process. To encourage this type of focus, I broke the experimentation roadmap template into multiple tabs where each tab maps to a roadmap for a specific growth metric — e.g. churn vs. reactivation vs. signups and so on.
Picking the right experiment to run is part art and science. By art I mean using judgement to craft a user experience worth testing. By science I’m referring to the practical constraints of testing new experiments on a relatively small population (i.e. sample size in statistics speak) when you’re still an early stage startup.
I often see startups try to run A/B tests in the same way that large companies like Google and Facebook do. They create a list of A/B test ideas that require fairly limited level of effort and then they start shipping dozens of small change tests fairly quickly. A classic example would be making changes to the call-to-action on a landing page, such as on the homepage, and perhaps testing the location of the call-to-action as well. The problem with this sort of test is that a startup often has a much smaller sample size (because they have less traffic or users of the product), so running and resolving that A/B test at high statistical confidence takes much, much longer than running a similar test at a high traffic product like Facebook. The relationship between experiment thoughtfulness and sample size is captured in the below diagram.
Here’s how to interpret it: Companies with a large sample size (a lot of traffic) don’t have to be as thoughtful with experiment selection and design. The reason is that the large company can make relatively small changes to the product, set up an A/B test to measure the effect, and then resolve the experiment in a matter of days at high statistical confidence because they have a wealth of data to lean on. On the other hand, a small startup with very little traffic (small sample size) needs to be much more thoughtful about experiment selection and design because an A/B test on a small sample size that produces a small change relative to the control will take weeks or months to harvest enough data to reach a statistically significant conclusion. I’ll demonstrate this effect in the below table.
Let’s imagine we have three different startups (A, B, and C — below). Each is going to run an A/B test on their homepage where the base conversion rate is 10%, the relative increase in conversion rate they are aiming for is 5%, leading to a new conversion rate of 10.5%. However, each startup has a different volume of daily traffic. Startup A receives 100 visits per day to the homepage, B receives 1,000 visits per day, and C receives 10,000 visits per day. Using the A/B testing calculator from AB Tastyto calculate the necessary test duration, we get the following results.
You can see from the data that the test duration declines significantly as a result of having more samples (i.e. traffic) in the test funnel. Now, let’s take a look at what happens when you tweak the magnitude of the relative experiment effect. In other words, when you run a test that produces a small, medium, or large change to the baseline conversion rate.
By increasing the magnitude of the relative experiment effect, the test duration declines precipitously. The key takeaway here is to aim for large changes. That seems like an obvious observation, yet I see many startups testing relatively minor changes to their product in the hopes it will produce a double digit increase in the target metric.
Finally, let’s look at what happens if we manipulate the base conversion rate. By base conversion rate I’m referring to the starting conversion rate. For example, if you have 100 visitors/day to your homepage and 1 user signs up, and you’re running an A/B test on the homepage, then you have a base conversion rate of 1%. If instead you run an A/B test midway through the sign up flow where there are 10 visitors per day, and 1 visitor manages to sign up at the end of the flow, then you have a 10% base conversion rate. What you’ll notice in the below scenario is that test duration decreases as a result of having a higher base conversion rate. Practically speaking, that means you’re more likely to reach statistical significance quicker if you A/B test in the bottom half of a funnel versus the top half since the bottom half has a higher base conversion rate.
To recap, there are a few key lessons to take away from the above scenarios:
It’s essential that anyone working on an experimentation team or roadmap understands the above statistical concepts. If so, they are less likely to stack their roadmap with poorly chosen A/B tests that will take too long to run and produce results too small to change the trajectory of the company.
Each year a batch of entrepreneurs set out to build the next great online community. Some attempt to build large horizontal platforms where users engage on topics ranging from immunotherapy to the Boston Celtics. Reddit would fit that description. Others seek to create vertical communities tailored to a particular subject and audience, such as Wheelwell for car enthusiasts.
There are many reasons to be on the prowl for the next great online community, either as an investor or operator. A leading reason is that the winners tend to be massive. Another reason is that successful online communities seem to grow perpetually through organic growth, which is the holy grail in startup land.
The unstoppable momentum of organic growth, driven by users creating new content on the platform, is what is often referred to as the “flywheel”. In technical terms, a flywheel is a device that stores energy. The more it’s revved up, the more energy it stores and the longer it can spin unaided. It sounds like magic, but there’s a simple explanation for it—at least as simple as physics goes.
What a flywheel does is it converts kinetic energy into potential energy. Kinetic energy is the energy that an object possesses due to its movement. Potential energy is the energy stored by the object due to its position. Archery provides a basic example. When you pull the bowstring back, you can say that the arrow has potential energy. And when it is released the arrow has kinetic energy.
Another key point about flywheels is that the bigger it is and the faster it spins, the more energy it stores and the longer it takes to slow down. Online communities that have established a “content flywheel” behave similarly.
Let’s take Reddit as an example. The stockpile of registered users is Reddit’s version of potential energy. When those users create content and the content is discovered in Google, shared via social media, or distributed online through other means, then the “arrow has been shot” so-to-speak. In this analogy, the user-generated content is kinetic energy. When new content is created it fetches new traffic and users into the platform, increasing the size of the flywheel and accelerating its rotational energy. It becomes self-propagating. And once that kicks in, good luck stopping it.
To put the power of a content flywheel in perspective, Reddit recently claimed 430M monthly active users. It’s 15 years old and still spreading its wings.
But how does one create such a platform driven by perpetual organic growth? Clearly, it can’t all be distilled down into a simple formula and bottled up in cans to be sold next to ketchup and mustard. It’s no commodity. There is no “secret sauce” that only Italian grandmothers and a few exceptional founders figure out. However, I do believe some of the ingredients are knowable and repeatable. This playbook will describe what those ingredients are, how they work, and what you can do about them in pursuit of building your own startup fueled by a flywheel.
I believe there are seven primary ingredients when it comes to building flywheels in software. Six of them are knowable and I’ll go into detail on each below. One ingredient is something only you, the founder, can figure out. It’s the “secret sauce” that makes your community stand out relative to the rest and is your unique innovation.
Here’s the full set of seven ingredients:
Let’s jump into each, how they work, and what you can do about them.
Let’s assume at this point that you, the founder, have already decided on the type of content community you want to build. It could be for scientists, sports enthusiasts, Chief Information Officers, or be the world’s next horizontal platform to compete with Reddit, Youtube, and so on. It doesn’t matter which option you’ve selected. What matters is you’ve vowed to create a dent in the content universe.
You begin toiling away in your preferred design tool with product prototypes, first starting with low-resolution concepts. After a bit of user testing, you’ve identified a variety of UX snafus, shuffled around the deck chairs a bit, and arrive at a prototype that’s ready for development.
The designs turn into an alpha. You test it with more users. The alpha becomes a beta. You test it with more users. Finally, you’re ready to launch it. You turn on the TV and play the iconic scene from Field of Dreams where the spirit of Shoeless Joe Jackson whispers, “If you build it, he will come.” And like ghosts emerging from a cornfield, users show up and engage with each other like long lost friends. Hours and hours of lively conversations are created and your community is flush with chatter.
Except, that’s not what happens. Conversations don’t spontaneously ignite and engagement is at a whisper. You’ve built it, but no one has come.
This is where your journey to creating the flywheel begins.Kevin Costner’s journey began with designing a field, but yours begins with designing your flywheel and hand-picking your early adopters.
Don’t get fancy. Start with 2,000-year-old technology and 500-year-old technology; paper and pencil. You don’t need modern software to design your first flywheel, so shut your laptop.
I believe there are four atomic units of a 1.0 flywheel:
Those elements represent the common building blocks of a content flywheel.
It begins with a visitor signing up to use the product. After a user has signed up, the user will gain access to the stockpile of content that exists in the application, which they may start consuming. Note that the stockpile of content won’t exist at first. I’ll get to that in the section about solving the cold start problem.
After consuming enough content, some users evolve into creators of content. The content the user creates leads to new traffic headed your way. A common example would be content indexed in a search engine or shared on social media, which fetches more traffic back to your community. Lastly, some of the newly harvested traffic will convert into newly acquired users that sign up to be a part of the community.
With a sketch similar to this, you can begin with a simple conceptual understanding of the content flywheel for your application. Assuming your product has launched and has at least a few hundred users, you can then measure the baseline conversion rates (CVR) at each step in the flywheel.
In the above example, the conversion rate (CVR) for the initial signup is 1.5%. Of the users that sign up, 20% of them go on to consume content in the application and 5% of consumers go on to become creators of content. The new content that is created then leads to new traffic generated for the application.
In this example, I chose the metric of visits per piece of content per month. A practical example would be a question answered on Quora or a thread posted on Reddit. In this case, each question on Quora or thread on Reddit would receive an average of 2 visits per month. Finally, the new traffic generated from the content created by new users leads to brand new users signing up for the product at a rate of 0.2%, which is a fairly common conversion rate for long-tail traffic coming from SEO.
The conversion rate to signup from this traffic is typically lower than traffic that goes directly to an application’s homepage, such as someone opting to go directly to Reddit.com and sign up. Visitors navigating directly to an app’s homepage have relatively high intent, likely because someone told them about the product, which is why the conversion rate is highest for direct homepage traffic.
And just like that, you have your first content flywheel designed and instrumented with empirical metrics. But your homework isn’t done yet. You’ve only completed the first of three assignments. And this one was the easiest since most online communities have a nearly identical 1.0 flywheel. In fact, you can just copy this flywheel and you’re off to a good start.
Tools in this chapter
TOOL |
Template
Product designer job descriptions
Examples of compelling product designer job descriptions for roles at Superhuman, Figma, and more.
Template
Product designer job descriptions
Examples of compelling product designer job descriptions for roles at Superhuman, Figma, and more.
TOOL |
Spreadsheet
Modern GTM Planner Calculator
This spreadsheet template will help you calculate the costs of marketing, sales, and demand gen to meet your revenue goals.
Spreadsheet
Modern GTM Planner Calculator
This spreadsheet template will help you calculate the costs of marketing, sales, and demand gen to meet your revenue goals.
TOOL |
Guide
The Unusual Salesforce (SFDC) Package
Installation and usage documentation for the Unusual Ventures Salesforce package.
Guide
The Unusual Salesforce (SFDC) Package
Installation and usage documentation for the Unusual Ventures Salesforce package.
TOOL |
Spreadsheet
Outreach Conversion Calculator
Track how well your outreach campaign is performing with this simple tool.
Spreadsheet
Outreach Conversion Calculator
Track how well your outreach campaign is performing with this simple tool.
TOOL |
Deck
Define your Market Segments
Define your market segments and get your GTM motion going.
Deck
Define your Market Segments
Define your market segments and get your GTM motion going.
TOOL |
Guide
Modern GTM User/Buyer Journey Map
USERs in our framework have a specific hierarchy of needs and a behavior pattern that founders must understand. In this map, we walk you through a USER's self-service approach to finding and using new solutions to the point where they are ready to engage with Sales. We then continue the journey as Sales engages the USER and BUYER to build a business case and close a deal.
Guide
Modern GTM User/Buyer Journey Map
USERs in our framework have a specific hierarchy of needs and a behavior pattern that founders must understand. In this map, we walk you through a USER's self-service approach to finding and using new solutions to the point where they are ready to engage with Sales. We then continue the journey as Sales engages the USER and BUYER to build a business case and close a deal.
TOOL |
Checklist
Pre-Seed gut-check questions
Questions to ask yourself and your potential co-founder(s), and about your product/market before you start your company
Checklist
Pre-Seed gut-check questions
Questions to ask yourself and your potential co-founder(s), and about your product/market before you start your company
TOOL |
Guide
Sales interview questions
Set of interview questions to vet for a successful VP of Sales or CRO hire
Guide
Sales interview questions
Set of interview questions to vet for a successful VP of Sales or CRO hire
TOOL |
Guide
Candidate intent assessment
Use this set of questions to assess candidate intent at each stage of the interview process.
Guide
Candidate intent assessment
Use this set of questions to assess candidate intent at each stage of the interview process.
TOOL |
Worksheet
Customer storytelling worksheet
Articulate the "Three Whys" for your company's B2B product
Worksheet
Customer storytelling worksheet
Articulate the "Three Whys" for your company's B2B product
TOOL |
Guide
Cohort Analysis Primer
Learn more about cohort analysis for consumer subscription businesses
Guide
Cohort Analysis Primer
Learn more about cohort analysis for consumer subscription businesses
TOOL |
Guide
Consumer Subscription Evaluation Rubric
Evaluation Rubric for a Seed-Stage Consumer Subscription Company
Guide
Consumer Subscription Evaluation Rubric
Evaluation Rubric for a Seed-Stage Consumer Subscription Company
TOOL |
Template
Product Testing Plan Doc
Share this document with your customer during your Product Testing Planning call.
Template
Product Testing Plan Doc
Share this document with your customer during your Product Testing Planning call.
TOOL |
Calculator
Deal Scorecard
Use the deal scorecard to determine the likelihood of a deal closing.
Calculator
Deal Scorecard
Use the deal scorecard to determine the likelihood of a deal closing.
TOOL |
Guide
Discovery Questions
List of common discovery questions. Feel free to use as a guide and customize for your startup.
Guide
Discovery Questions
List of common discovery questions. Feel free to use as a guide and customize for your startup.
TOOL |
Worksheet
Hypothesis & Personas Exercise
Come up with your own hypotheses and personas
Worksheet
Hypothesis & Personas Exercise
Come up with your own hypotheses and personas
TOOL |
Guide
Employee guide template
Use Ride Report's employee guide as a starting template for creating your company's own guide.
Guide
Employee guide template
Use Ride Report's employee guide as a starting template for creating your company's own guide.
TOOL |
Guide
ICP and Personas Exercise
Questions to define your ICP and User and Buyer Personas
Guide
ICP and Personas Exercise
Questions to define your ICP and User and Buyer Personas
TOOL |
Checklist
Company/candidate checklist
A comparative checklist for companies vs. candidates
Checklist
Company/candidate checklist
A comparative checklist for companies vs. candidates
TOOL |
Guide
Customer Interview Guide
Guide for interviewing customers
Guide
Customer Interview Guide
Guide for interviewing customers
TOOL |
Worksheet
Brand Story Exercise
Messaging checklist and exercise to build your own brand story
Worksheet
Brand Story Exercise
Messaging checklist and exercise to build your own brand story
TOOL |
Worksheet
Get to Aha! DNA Test
Test to discover your brand's DNA
Worksheet
Get to Aha! DNA Test
Test to discover your brand's DNA
TOOL |
Template
Offer Letter Template
Offer letter template for early employees
Template
Offer Letter Template
Offer letter template for early employees
TOOL |
Worksheet
"Why Buy Now" Framework
Framework for discovering your "Why Buy Now"
Worksheet
"Why Buy Now" Framework
Framework for discovering your "Why Buy Now"
TOOL |
Three Why's Framework
Framework for discovering "Why Buy Now"
Three Why's Framework
Framework for discovering "Why Buy Now"
TOOL |
Guide
Customer Research Guide
Guide to Conducting Customer Research
Guide
Customer Research Guide
Guide to Conducting Customer Research
TOOL |
Guide
Product Requirements Template
Product Requirements Template
Guide
Product Requirements Template
Product Requirements Template
TOOL |
Template
Amazon Press Release Template
Amazon Press Release Template
Template
Amazon Press Release Template
Amazon Press Release Template
TOOL |
Deck
Project Review Template
Project Review Template
Deck
Project Review Template
Project Review Template
TOOL |
Deck
Kickoff Meeting Template
Kickoff Meeting Template
Deck
Kickoff Meeting Template
Kickoff Meeting Template
TOOL |
Guide
Consumer Social Evaluation Rubric
An evaluation guide for pitching your consumer social app
Guide
Consumer Social Evaluation Rubric
An evaluation guide for pitching your consumer social app
TOOL |
Deck
Outline Seed Deck
Outline investor deck for a Seed Round
Deck
Outline Seed Deck
Outline investor deck for a Seed Round
TOOL |
Deck
Outline Series A Deck
Outline investor deck for a Series A Round
Deck
Outline Series A Deck
Outline investor deck for a Series A Round
TOOL |
Deck
Outline Series A board meeting deck template
Outline Series A Board meeting presentation
Deck
Outline Series A board meeting deck template
Outline Series A Board meeting presentation
TOOL |
Template
Executive memo template
Simple template for aligning executive staff
Template
Executive memo template
Simple template for aligning executive staff
TOOL |
Spreadsheet
Founder prioritization heatmap template
GPP Heatmap to sort and tackle competing priorities
Spreadsheet
Founder prioritization heatmap template
GPP Heatmap to sort and tackle competing priorities
TOOL |
Filter
Deal Qualification Filter
Filter to qualify sales leads and discover real opportunity
Filter
Deal Qualification Filter
Filter to qualify sales leads and discover real opportunity
TOOL |
Calculator
Math of Sales Calculator
Calcuate needed outreach to reach revenue goals
Calculator
Math of Sales Calculator
Calcuate needed outreach to reach revenue goals
TOOL |
Deck
Outline customer deck
Outline deck for your first customer meeting
Deck
Outline customer deck
Outline deck for your first customer meeting
TOOL |
Worksheet
Product Roadmap 4 lists worksheet
Exercise for defining your four lists
Worksheet
Product Roadmap 4 lists worksheet
Exercise for defining your four lists
TOOL |
Worksheet
Focus exercise for SaaS startups
Exercise for finding your focus
Worksheet
Focus exercise for SaaS startups
Exercise for finding your focus
Explore product testing techniques, designing an organization for growth, and creating content communities to help you gain traction with users.