From sensitivity to simulation
Deciding under uncertainty when every variable changes
Most business cases ask whether some new option beats the old one. The harder, more relevant question is whether the decision actually makes money once the whole system varies. Many marketers have seen this, someone brings in a fancy new top of the funnel technique after A/B testing, which doubles lead volume only for those leads it produced to only be such a low quality that any extra “benefit” is simply written off. Conversely, sometimes a new system that delivers a smaller lift than we’d hoped gets written off, because we haven’t counted the downstream condition variation that could still make it profitable.
In the previous article (linked below), we explored the decision-making scenario of a new MQL conversion tool from a vendor which showed genuine potential uplift. The issue was that, given the sample size, we were not sure that uplift was true and how much this would hold across time. The worked example showed a non-zero risk of loss across several years into the future. A key issue was that it treated only the MQL conversion rate as something that could vary and everything else from the average profit per customer to the SQL->close conversion rate as fixed. This article explores the idea that these can also vary in the real world and looks at what this might do to your final decision.
When the equation has too many moving parts
There exists a scenario where the new vendor solution does work well but downstream metrics like leads or profit collapses naturally through statistical variation, causing the benefits to disappear. So, even if the vendor in the previous article’s example delivers everything promised, the case can still be net negative.
This is the bigger problem the previous article skipped. We were only asking “is the vendor better than the existing setup?”, a comparative question. But the actual decision is absolute: “will signing this contract make us money once we account for the whole system?” The comparative question is easier, downstream parameters were ignored. The absolute question doesn’t ignore them. Lead volume, close rates, average deal size, these all affect whether the vendor contract pays for itself, regardless of whether the vendor’s lift is real. All these downstream metrics vary and the model must take into account the variability of the whole system.
When algebra gives up, simulation begins
Problems like this quickly become analytically intractable. Once uncertainty exists at multiple stages of a system, lead volume, conversion rates, deal size, retention, costs, the number of possible outcomes explodes. Rather than solving these systems exactly with algebra, analysts increasingly rely on simulation methods, the most influential of which is the Monte Carlo method. Developed in the 1940s by scientists working on the Manhattan Project, it works through repeated random sampling to approximate solutions to problems too complex to solve directly. This allows for 2 things:
· Estimate ranges of uncertainty
· Propagation through systems of uncertainty
Estimate ranges of uncertainty
A lot of marketing analytics uses what we call point estimates. For example, in the previous article’s example, the MQL conversion rate was 59 out of 1234 = 0.048 converted to SQL for the existing tooling and 39 out of 594 = 0.066 for the new vendor tool. These are point estimates in the sense that we don’t know how well these ranges hold over time, they may not be the ‘true’ long-term conversion rates. We just know they were the conversion rates for that sample.
To get a further intuition for this, let’s consider 2 scenarios at the extreme opposite ends. An A/B test for MQL -> SQL shows that A group converted 10 out of 100 = 0.1 emails sent, and B group converted 800 out of 10,000 = 0.08 emails sent. A group has a higher conversion rate but how can we be sure we are not observing noise due to the small size of the data?
We can run a Monte Carlo simulation for these 2 examples to demonstrate how this looks. The below shows what the uncertainty, or likelihood, of both groups are. We can see clearly that, although the bigger observed conversion rate is with A group, the uncertainty is far wider than B group, this is shown by how tall-and-narrow or wide-and-flat each density curve is.
Taking this further, we can actually calculate the actual probability of A > B and plot what this looks like. The plot below shows the likely range of uplift, with the bulk of the density plot on the right-hand of zero. In fact, we can calculate how much of the curve lies on the right-hand size of zero and this gives us a 73% chance that A > B. Read in the negative sense, there is a 27% chance that B wins.
Propagation through systems of uncertainty
The second benefit of Monte Carlo simulation is that it allows us is to show how uncertainty propagates through a system of such distributions.
To illustrate this, imagine these 2 scenarios:
Scenario A
5 out of 50 leads convert from MQL to SQL;
6 out of 20 SQLs close
Scenario B
200 out of 2,000 leads convert from MQL to SQL;
240 out of 800 SQLs close
If you check the maths on this the actual observed conversion rates don’t change between scenario A and B, they both share an MQL to SQL conversion rate of 10%, and an SQL to close conversion rate of 30%. Therefore they both produce the same MQL to close rate of 3%. The key difference is the uncertainty how that uncertainty propagates through the chain from the MQL conversion rate to the final close rate. The below plot illustrates this clearly. On the left, in red, Scenario A shows the conversion rates from small samples, the wide, flat shapes mean we're far from certain where the true rates actually sit. On the right, in blue, Scenario B shows the same observed conversion rates but from much larger samples, producing narrow, confident peaks. At the bottom, when we multiply each scenario's two stages together to get the full MQL→close rate, that uncertainty carries through: Scenario B remains a tight spike, while Scenario A spreads even wider, despite both scenarios pointing at the same headline number of 3%.
What does this mean for the previous article’s example where encountered the profit equation. Recount that this allowed us to predicted expected uplift in profit at any given years in the future?
Where:
Pn = Profit at n years into the future
MQLv = the vendor’s MQL conversion rate
MQLE = your organisation’s existing conversion rate
Pc = Average profit per customer
L = Average number of leads each year
SQL = The average SQL conversion rate
Ca = Annual cost of the new vendor system
Yn = n years into the future
Ci = Initial cost of the new vendor system
The Monte Carlo method allows us to treat all of these with uncertainty. We don’t know that the profit is always 16,000, it’s just the average profit. We also don’t know that the number of leads entering the top of the funnel is always going to be 1,300 just like we cannot know that the SQL->closed conversion rate is always going to be 0.22. The previous article treated all these as fixed. Monte Carlo simulations allow us to put variation values across the whole equation and such variation can be obtain from past data.
It could be that when the MQL lift turns out low, the SQL->close rate might be high. When leads are down, the contract value might be up. Things move in different directions. The chance that everything goes badly at the same time is much smaller than the chance any one thing goes badly on its own.
A negative lift can’t be rescued. The formula computes incremental profit, so if the vendor underperforms the existing system outright, high close rates and high contract values amplify the loss, those same conditions would have generated more revenue under the system you’re replacing. But a small positive lift, below the trial estimate, can still be rescued: above-average close rates or contract values turn a marginal uplift into a profitable one.
When you let the whole system vary together which is what Monte Carlo allows, the realistic downside at year 5 –72,827 € is much milder than the worst-case-on-one-input version of –211167 € explored in the previous article.
Notice also that the probability of loss falls from 17% at year one to 9% at year five, and then stops. Longer horizons fix the implementation-cost recoupment problem, but they don’t fix the bad-uplift scenarios. If the vendor’s true lift is small enough, you’re underwater forever. Same data. Same equation. A more honest answer.
Next time you need to make a long-term business decision, ask two questions. What is the probability we lose money? And which input, if measured more precisely, would shrink that number the most?
Previous article: