Explore, Experiment, Invest

The history of Silicon Valley is strewn with failed products that, in hindsight, were built with unwarranted confidence. Some of these products were simply before their time (Apple Newton, Webvan, Pets.com) while others were chosen based on deeply flawed information (Google+, Juicero, Facebook’s Metaverse, everything about cryptocurrency). It’s tempting to praise technologically impressive yet poorly-timed products but when a company with finite capital is early or late to market it’s just as bad as being completely wrong.

Every healthy product initiative develops through three distinct stages. Confusing them is, in my experience, the primary source of product failures. Getting them right gives you the best chance to build something of value within your budget.

These three stages always happen in order, and skipping any one of them means your strategy now relies on luck.

In case you can’t tell from this whole book, I love working at startups. There’s something magical about creating something from absolutely nothing. A team at a startup solves problems by inventing solutions quickly, one of the more thrilling things one can do in a creative field.

Sometimes the team invents just the right thing, succeeds at selling it, and everything goes right for them. They explore their market carefully, diagnosing potential customer problems clearly and correctly. They run exactly as many tests on their new product as they have financial runway to perform. They rapidly iterate through the phases of data-gathering, hypothesis testing, and then delivery. They don’t get beaten by a competitor, the market does what they expect, and they aren’t surprised by the difficulty of the implementation.

More often the whole endeavor fails miserably. Even when they do everything right it’s hard to win at a startup. And, if we’re honest, most strategies and the implementations of them are quite poor.

The process of Explore, Experiment, Invest maximizes the chances a product – big or small – will succeed. It’s a particularly good process for a team to use when constructing their first product, when capital is most scarce. Yet I find that the first product is the one time where many founders apply decent rigor. It’s the second product that they tend to get sloppy with. Particularly if the second product serves primarily to meet the CEO’s personal hopes instead of diagnosed users’ needs.

Companies that succeed with their first product because of market timing or pure luck are particularly susceptible to skipping steps in Explore, Experiment, Invest. If your product gained rapid adoption immediately you might assume that the next product you create will do the same. It’s like they say in Vegas about gambling addiction: The worst thing that can happen to you is to win big early. You’ll spend the rest of your life trying to recreate that initial experience.

In the same way, engineering leaders who’ve mostly been at high-growth companies may assume that products mostly just work out. That headcount always grows. That the hard part of the job is onboarding and organizing all these people. But once the money dries up they need to immediately figure out which of their ongoing projects are showing any actual promise in the marketplace. As soon as budget matters, it’s important to know whether the thing you’re building is moving forward.

We must be rigorous when we craft features and products that we expect strangers to value. We can’t guarantee our own success but we can maximize the chances of it by simply being honest with ourselves about what we do and don’t know.

Every feature under development is somewhere along the path of exploring, experimenting, and investing. Each stage offers a different technical and shipping context for the engineers who work on it as well as the product leaders who are guiding it.

Let’s go through those three stages of process so you can identify when you’re in one or when you’ve accidentally skipped past it

Explore – Step #1

We’re in the Explore phase whenever when we’ve identified a problem but have no evidence that any specific solution will work. Here, the goal is to get as much data as possible as cheaply as possible so we can form hypotheses. This might include creating a proof of concept or a working prototype, it might be asking colleagues what they think or what they’ve seen before, and it might be analyzing data available to us.

The goal here is not to build a solution, but to form coherent hypotheses. To go from “I wonder what might work?” to “I believe this specific thing might work.”

This is where startups exist by default.

How to know if you’re in the Explore phase?

The output of a successful exploration is user research, technical prototyping, synthetic market tests (like running an ad campaign that points to an email-harvesting page), and letters of intent from prospective business development contacts.

Without these it’s difficult to make meaningful hypotheses in the form of “we can build feature X and then sell it to our users via this sales/marketing channel”

If a statement in that form feels unsupported by any data at your company, consider halting development plans and continuing Explore work until that data exists.

How companies try to skip the Explore phase

It’s so easy to skip this stage. You may hear a colleague (or yourself) say “Let’s run an experiment to see if users will give us their mailing addresses”. That’s not an experiment, it’s an activity. It’s merely exploring what happens when you build an address-collecting form. What’s the decision that happens as a result of, say, 10% users submitting addresses? 50%? 99%?

The key difference between an exploration and an experiment is that explorations cannot fail. They’re an attempt to get data. Even one that results in zero data is a successful exploration if it can dissuade the team from further work that would have been useless.

If this were an actual experiment it would sound like “Let’s test whether asking for mailing addresses, sending printed materials to our users, and then trying to re-engage them online results in net increased revenue.” The answer to this is either “yes”, “no”, or “we screwed up the experiment and here’s a meeting to figure out why.”

Experiment – Step #2

Once we have enough data to make credible proposals for a solution, we try to prove or disprove whether that solution is viable. We do this by creating low-cost experiments. Hopefully, our assumptions are right and we can continue to work on the experiment until it turns into a production-grade solution.

But if our assumptions are wrong, we’ll have avoided spending orders of magnitude too much work on a product or solution that was under-informed.

Note: At some companies the word ’experiment’ has a very narrow meaning, typically relating to growth experiments. The part of the product that tries to convert users undergoes frequent testing in UI variations, mostly in email marketing and landing pages.

That same experimentation may not happen across the rest of the product, but a broader experimentation is the only way to know if the overall market actually exists, not just whether we can convert sales within it.

A good experiment takes the form of a falsifiable hypothesis. It’s a statement about what you think might be true, phrased clearly enough that it’s possible for it to be false.

Well-articulated experiments sound like this:

• A user is more likely to re-engage within 2 months if we reach out to them daily than if we do it weekly
• We can charge 50% higher prices for our current enterprise customers with low enough churn to make this a net increase in revenue
• We can increase home page conversions by more than 10% if the pages perform twice as fast on mobile as they do today

To better understand the components of a good experimental thesis let’s first look at what makes a bad one. In their book “Superforecasting” Philip E. Tetlock and Dan Gardner summarize the cognitive trap of believing in one’s hypothesis:

[Scientists] know that no matter how tempting it is to anoint a pet hypothesis as The Truth, alternative explanations must get a hearing. And they must seriously consider the possibility that their initial hunch is wrong. In fact, in science, the best evidence that a hypothesis is true is often an experiment designed to prove the hypothesis is false, but which fails to do so. Scientists must be able to answer the question “What would convince me I am wrong?” If they can’t, it’s a sign they have grown too attached to their beliefs.

Tetlock, Philip E.; Gardner, Dan. Superforecasting (p. 38). Crown. Kindle Edition.

A bad experiment is anything that can’t dissuade us from our beliefs. For example, let’s imagine we removed some of the success criteria from the examples above:

“A user is more likely to reengage if we reach out to them daily then monthly”

Note the lack of time frame for the reengagement. The more someone believes this the longer they’ll ask to run the experiment, running down the clock on your company’s runway and using up time that could be spent on other experiments.

“We can charge higher prices for our current enterprise customers without losing many”

What is “higher”? What is “many”? The inverse of this sentence is also probably true because it’s so vague.

“We can increase home page conversions by more than 10% if the pages are much faster”

How much faster? And by when, compared to what baseline? This could be an infinite workstream with no material gains.

The point of an experiment is to find out if our beliefs are incorrect before using them for financial or staffing investments.

How companies try to skip the Experiment phase

Bypassing this phase is super common for companies looking to create their second product. If they have a successful flagship product they may have spent so long enjoying that success that it’s hard to remember the enormous likelihood of failure for even very well-made products.

Failure Mode: The product that meets only the CEO’s needs

There’s a specific, very dangerous form of this that I’ve seen several times: A CEO staffs a new product initiative, declaring that it will be a success.

That might sound ludicrous but it’s quite common. Maybe the CEO failed to satisfy questions at a board meeting about how the company will expand into new markets. Maybe the CEO is bored with the status quo and wants to feel the energy they remember from when the company was younger. For whatever reason, the CEO (or another feedback-immune leader) staffs a team to work on a solution. The problem they’re solving is “The CEO wants this product built.”

The team working on this pet initiative will be in the Invest stage right from the start. They need to figure out what market they’re in – and if that market even exists – while also working toward a specific implementation. Not only is this super failure prone but it’s extremely stressful for the team. The whole company is watching the CEO cheerlead you while you struggle against a harsh reality: Customers pay money to solve their needs, not your CEO’s.

Depending on the length of runway given to this team it may take the company months, years, or decades to recognize the team’s struggles as a sign of a failed product. The budget and runway will exist for as long as senior leadership wants the product to succeed. And as long as they want it to succeed the less they’ll be open to critical feedback.

At the time of this writing both Jeff Bezos’ Alexa and Mark Zuckerberg’s Metaverse have finally felt their leashes go taught as reality caught up to an overfunded, unrealistic, CEO-driven product strategy that bypassed the critical first two steps.

Failure Mode: The Startup Within a Startup

I should mention, considering the audience of this book, that one of the tempting paths toward an executive title is to join a big, slow company that’s trying to inject new life in itself. They may offer you a VP of Engineering title and ask you to lead a startup within the company – getting the flexibility of a startup with the resources of the larger firm. Setting aside that that’s never how it works (you get the red tape of the larger firm and too much scrutiny to make necessary mistakes), by the time they’re talking to you they likely have both a problem they’ve identified and a solution they have in mind.

You’ll need to determine whether they have skipped steps in this Explore, Experiment, Invest process. Have they imagined a solution they like that you might spend years building only to have it flop miserably? Or do they have good research, a healthy sense of the market, and some hypotheses that they’ve been able to prove or disprove?

I recommend you don’t accept jobs like this. But if you do, make sure you negotiate for the autonomy and resources necessary to scrap the entire project and reimagine a solution to the identified problem. Because if the investment thesis isn’t bulletproof then success in your go-to-market will depend on luck.

Invest – Step #3

This is the fun part. If we’ve completed enough exploration to form hypotheses and we’ve completed enough experimentation to know which of those hypotheses we believe then we can make a real plan.

The goal in this phase is to be clear about what resources we expect to pay (usually in staffing and time) and what we expect to get back, by when. The clearer we can be about this the easier it’ll be to see where we still have holes in our information. We’ll never have perfect knowledge, but by identifying the gaps we can lean away from depending on those areas too much.

A good investment thesis can be quite high level, if we trust the underlying knowledge. It can serve as a one-liner that you reference in every all-hands meeting.

Good investment theses look something like this:

• The merchant team will spend the next 2 quarters improving API completeness and we expect more than double the amount of 3^rd^ party signups over the following year
• We’re building a white-label version of our product that we expect to sell within a year to most of the 600 customers who signed a letter of intent
• We’re pivoting to a B2C model where we expect at least 10,000 users subscribed in year one at a $12/month price

These statements don’t mention how or why the company has chosen this path but there should be plentiful data from the Explore and Experiment work you can share with curious colleagues. But you don’t need to explain all that every time you state the investment – you can trust yourself and your team when you say you’re doing X and you expect Y to happen.

How companies try to skip the Invest phase

A tragic amount of work at early-stage startups is mere activity. Many of the major initiatives I’ve seen at the Seed and Series A companies I’ve advised has been in the form of “we built this because it’s cool and people will love it.” That worked out for Instagram, but I personally recommend more rigor.

Conclusion With a Caveat

None of the above should be taken as an excuse to halt development and wait for perfect confidence. If you find yourself or a colleague providing consistent pushback on any work in the name of “we don’t know enough yet” there is likely a relational problem that needs to be addressed. Not because there’s a right way to apply these tools but precisely the opposite: There’s no correct amount of confidence to reach before moving forward. What you, your CEO, and your peers need to agree on is how much uncertainty is acceptable. How much product risk are you willing to accept as a tradeoff for the risk of waiting longer, of getting more certainty.

There’s no right answer. But if you’re aligned with your team – and you’ve gone through at least some of the work in each of these phases – then you’re likely doing better than your competitors. Which, in a startup, is often all that matters.

Applying Explore, Experiment, Invest to Product Development

A team that’s working on a product in Explore has a totally different working style from one working on a product in Invest. The things they need from the rest of engineering are different, the planning is different, and in my experience the personalities that thrive on that team will be different.

With this model of product risk stage in hand we can combine all 3 dimensions into the Product Development Cube and develop an informed team process. Read Executive Engineering to learn more