Avoid wasting time testing inconsequential stuff when growing a product

Artwork by Justine

Traditionally, marketing focuses on acquisition as the main growth lever. The problem is if you don’t activate new customers, they’re unlikely to retain long-term. In SaaS businesses especially, if you don’t retain customers then you’re really just replacing a customer each month. You might be able to do it effectively for a few months. But over time, it’s an unsustainable way to grow a business.

Begin with a north star

The best way to start thinking about this process is to begin with your core action, this is the thing people do that drives value for them. This is sometimes called a north star metric. The north star metric is one of the breakthroughs that came out of Facebook and now all the effective growth companies have latched onto the concept.

All that matters is that your north star metric reflects the value that is delivered to users. You’re trying to track the aggregation of customer value over time. The idea is that customer value is what drives retention and retention is what drives sustainable revenue.

A couple of examples here would be Airbnb’s north star is ‘nights booked’. So whether you’re a host or a guest, when a night gets booked, everyone is getting what they came to the platform for. The more nights booked over time, the more value is being created in the system, and that value is what drives sustainable growth.

Then find high leverage opportunities

Once you’ve narrowed in on a north start and began tracking your core action, the next step is to understand the relationships between the variables that move your north star. The only way to do this is to test what actually drives growth.

There are two different types of testing. The first is testing to discover and the second. is testing to optimize. The best way to think about this is like a game of battleship.

If you’ve never played battleships then here’s a 2-minute walkthrough of how the game works:

The point is that when if you get the coordinates wrong, you still learn something. Over time, you start to get an idea of where those opportunities are. This kind of exploratory ping is a discovery test. When you do hit upon a growth opportunity, it’s safe to assume there’s a better way to do things. The only way to find a better way is to start testing variations. These kind of tests are optimization tests.

I first heard this Battleship example in a talk by Sean Ellis. He claimed to have stolen it from Brian Balfour so I think it’s safe to steal once again because it’s such a good analogy.

The general rule here is that the more testing you do, the faster you tend to grow. If you test 10 things, the chances of finding a growth opportunity are higher than if you take one or two shots. However, that’s not an excuse to test pointless details. Tiny changes like the shade of a button are a waste of time because you could be using your limited resources to test more consequential stuff.

The way that you find leverage is that you look at the data, and study where you’re losing people. For example, if in a funnel you find that you have a 90% drop-off somewhere, running tests around how to fix that are going to be more consequential to growth than testing different colors of your sign-up button.

Then use qualitative data to brainstorm why

Quantitative data helps you narrow in on high leverage opportunities. Once you’ve found a focus the next step is to figure out what the problem is. Qualitative data is much more useful here. Dress it up however you want but fixing product problems is a guessing game. Surveys, session recordings, competitive research, industry best practices, and (my favorite) customer interviews make it an educated guessing game.

If you want to rely on intuition alone then you look at the problem and rely on common sense to figure out what the most likely cause of the problem is. A more effective approach is to get as many relevant people in the room as you can and leverage as many different viewpoints as possible. You want diversity, so rather than getting the whole product team into a room, aim for one person from sales, one from customer support, an analyst, a developer, and someone from the customer research team.

These kinds of group brainstorming sessions can fall apart easily. It’s important to make sure that everyone understands the opportunity you are focused on. This is not a general-product-feedback session.

Typically, people like to start by throwing solutions on the board. This makes it really hard to compare and prioritize different solutions. An easier approach is for everyone to come up with ideas around why the problem exists first.

Once you have a shortlist of the most likely problems then you want to come up with the simplest solution to test if the problem exists. Don’t spend 3 months building a possible solution to something that might be a problem. Spend a month building 10 simple solutions to the most likely problems. Then spend the remaining 2 months building better solutions to the problems that had a measurable impact on your north star metric.

Identify and focus on the highest leverage opportunity you have. Then run as many discovery tests as your resources allow using the best guesses you can source.

As soon as you hit a battleship, switch to optimization tests and figure out the best way to execute that solution. Measure success in terms of improvements to your north star metric. That’s why it’s so important to make sure this number corresponds to actual user value when you begin.

After a while, you’ll start hitting a ceiling on the optimizations you can make. This is when it’s time to switch focus and repeat the process with the next high leverage opportunity.

Testing takes a lot of expertise to set up properly and a lot of time to run effectively. This is why it’s important to avoid wasting time testing inconsequential stuff when growing a product. You want to focus on high leverage opportunities and systematically test your best assumptions before you narrow in on optimization tests. The more tests you run, the more you will learn, and the more likely you are to drive a successful growth outcome.