In product, we encounter many types of risks, and managing them is one of our most important tasks. While there are many different risks to consider, they’re going to fall into two essential categories when deciding how to manage them.
When you identify a risk, first ask yourself: “can we figure this out in advance?”
If you can, then your focus is on mitigating that “can know” risk. For a “can’t know” risk, your best tool is a well-designed experiment.
For a “can know” risk, your next question is: “Should you figure it out in advance?” While you won’t know precisely how hard it will be to figure out in advance, get a rough sizing and compare it to the risk’s potential negative impact. If the potential fallout from your risk is reasonably small and your effort to mitigate the risk is high, it might not be worth the extra effort. You’ll also want to think about probability. If it’s possible but improbable, you’ll want to weigh your decision accordingly.
Suppose you’ve got a risk that you can’t uncover in advance (often related to user behavior). In that case, you’ll want to ask yourself a similar question — “is it worth validating with an experiment?”.
Your risk experiment will either focus on mitigating and learning or just on learning. A mitigate & learn experiment will reduce the surface area of risk, e.g., rolling out a test feature to 1% of users. A learning-only experiment will focus on gaining insight and will usually be a typical A/B split test.
If your “can’t know” risk has a low impact (or a low impact when weighted by probability), consider not running the experiment or opting for a learning-only experiment.