Summary
Not every experiment needs to win. I know that’s not what the board wants to hear. It’s not what investors expect. It’s not what gets celebrated in all-hands meetings or written up in case studies. But it’s the truth that separates teams that compound learning from teams that just run campaigns.
Some experiments exist to scale. Others exist to reveal what the next smart bet looks like. And if you can’t tell the difference, if you’re only willing to try things you know will work, you’ll never find the breakthrough that changes everything.
Because here’s what 25 years has taught me: the secret to growth isn’t perfection. It’s iteration that compounds. And compounding requires something most organizations are terrible at: learning from things that don’t work.
The Tyranny of Scale
Every startup reaches a moment where “will it scale” becomes the filter for every decision. Someone proposes an experiment. The first question is always: “But does it scale?” Can we do this for a thousand customers? Ten thousand? A hundred thousand?
If the answer is no, the idea gets killed. Too manual. Too expensive. Too dependent on things we can’t automate.
I’ve been in those rooms. I’ve asked that question myself. And I’ve watched brilliant ideas die because they failed the scale test before they passed the learning test.
Here’s what we miss when we do that: The things that don’t scale are often the only way to discover what will.
You can’t A/B test your way to a new category. You can’t automate your way to product-market fit. You can’t scale what you don’t understand yet. Understanding comes from doing things that are inefficient, manual, and unrepeatable at first.
Paul Graham wrote about this in 2013: “Do Things That Don’t Scale.” The essay became gospel in Silicon Valley. Everyone quoted it. But most people only remembered the tactics: hand-recruit your first users, deliver exceptional service, manually onboard every customer.
They missed the deeper point: unscalable experiments are how you build the knowledge that makes scaling possible. You don’t start with a machine. You start with a hypothesis, tested manually, refined through iteration, then systematized once you understand what actually matters.
The unscalable phase isn’t a compromise. It’s the laboratory.
What Failure Actually Teaches
I spent two years at a B2B SaaS company where we ran 40+ growth experiments. Seven succeeded at scale. Twelve showed promise but couldn’t get past unit economics. Twenty-one failed completely.
Most teams would call that a terrible hit rate. We called it a knowledge factory. Because here’s what the failures taught us:
The messaging test that bombed revealed that our ICP didn’t respond to efficiency narratives, they responded to risk mitigation. We’d been selling speed when they were buying safety. That insight changed our entire positioning and tripled our deal flow.
The partnership channel that flopped showed us our product wasn’t a vitamin for that segment, it was trying to solve a problem they didn’t know they had. We killed that segment from our targeting entirely and reallocated budget to where real pain existed.
The content experiment that got zero engagement taught us we were writing for the persona we thought we had, not the one actually buying. We rebuilt our entire content strategy around the language that was working in sales calls instead of the language that sounded good in marketing reviews.
The product-led motion that stalled revealed a fundamental gap in our onboarding; users understood what the product did but not when to use it. That insight led to a complete redesign of our activation experience and cut time-to-value by half.
None of those experiments “worked.” But every single one taught us something that made the next experiment smarter. That’s what most organizations don’t understand: failure isn’t the opposite of success. It’s the raw material for it. But only if you know how to extract the lesson.
The Anatomy of a Good Experiment
Not all experiments are created equal. Some fail and teach you nothing. They’re just noise: poorly designed, under-resourced, abandoned before you learn anything meaningful. Others fail and transform your strategy. They reveal assumptions you didn’t know you were making. They expose gaps in your understanding. They point you toward the breakthrough you weren’t looking for. The difference is design.
A good experiment, whether it succeeds or fails, has four characteristics:
- A falsifiable hypothesis
Not: “Let’s try content marketing.”
But: “If we publish weekly case studies focused on risk mitigation, we’ll see a 20% increase in enterprise demo requests within 60 days.”
That’s falsifiable. You’ll know if it’s true or false. And when it’s false, you’ll know which part was wrong; the channel, the frequency, the message, the metric, or the timeline.
Vague experiments generate vague learnings. Specific hypotheses generate actionable insights.
- Clear success criteria and clear learning criteria
Most teams define success: “We’ll know this works if we hit X metric.” But they never define learning: “If we don’t hit X, we’ll know whether the problem was awareness, messaging, or conversion by measuring Y and Z.”
You need both. Because sometimes an experiment fails but the failure teaches you exactly what to try next.
I worked with a team that tested a freemium motion. It failed spectacularly; almost no free users converted to paid. But they’d instrumented the experiment well enough to see why: free users were getting value, but hitting friction at a specific feature gate. The freemium motion didn’t work, but it revealed exactly where value perception broke down. They killed freemium. But they used the insight to redesign their trial experience, which became their highest-converting channel.
The experiment failed. The learning compounded.
- Enough time to generate signal
Most experiments don’t fail. They get killed before they have a chance to teach anything. Someone launches a new channel, gives it three weeks, sees weak results, declares it “not working,” and moves on. But three weeks isn’t enough time to learn whether the channel is wrong or whether your execution needs refinement. You haven’t separated signal from noise yet.
Good experiments have pre-determined time horizons: “We’ll run this for 90 days, then evaluate whether we’ve learned enough to make a decision.”
Sometimes the learning comes early. Sometimes it takes the full duration. But you commit to the learning period, not just the launch.
- Documented outcomes—win or lose
This is where most companies fail.
Successful experiments get celebrated. Case studies get written. Team meetings happen. The knowledge gets encoded.
Failed experiments get buried. Someone mentions them once in a retro, everyone nods, and the learnings evaporate. Six months later, a different team tries the same thing and learns the same lesson all over again.
That’s not iteration. That’s repetition. If you want learning to compound, you need to document failures with the same rigor you document wins.
What was the hypothesis? What did we see? What did we learn? What should we try differently? What should we never try again?
Build a knowledge base of failed experiments. Make it searchable. Make it required reading for new team members. Because the goal isn’t to avoid failure. It’s to avoid repeating failures.
The Culture That Enables Experimentation
I’ve worked with hundreds of teams. The ones that build real innovation cultures all have the same characteristics and they’re not what you’d expect. It’s not about “psychological safety” or “fail fast” posters or giving people permission to experiment. It’s about three specific cultural patterns:
- Experiments are funded differently than operations
Most companies have one budget. Every dollar spent on experiments competes with dollars spent on proven channels. So the CFO looks at the spreadsheet and says: “Why are we spending money on this unproven thing when we could put it into paid search, which has a known ROI?” The experiment loses. Every time.
The teams that innovate consistently separate their budgets. They allocate X% to operations (running what works) and Y% to experiments (discovering what’s next). The experimental budget has different success criteria: learning, not immediate ROI.
That gives teams permission to try things that might not work, because the expectation isn’t that every experiment scales. It’s that the portfolio of experiments generates enough learning to find the next scalable thing.
- Failure is interrogated, not punished
Weak cultures punish failure. People get blamed. Careers stall. Everyone learns to play it safe. Strong cultures interrogate failure. When something doesn’t work, the team gathers and asks:
- What did we expect to happen?
- What actually happened?
- What does that tell us about our assumptions?
- What should we try next?
- What should we stop trying?
No blame. No defensiveness. Just disciplined pattern recognition.
I worked with a VP of Marketing who ran a monthly “Failure Debrief.” Anyone who’d run an experiment that didn’t hit its goals presented to the team. Not to get criticized. To teach.
They walked through the hypothesis, the execution, the data, and the learning. The team asked questions. Everyone left smarter.
That single ritual did more to build innovation culture than any amount of talk about “embracing failure.”
- Career advancement rewards learning velocity, not just winning
If you only promote people who hit their numbers, you’ll build a culture of risk aversion. Because the rational move is to do what’s safe. Run the proven playbook. Hit your target. Get promoted.
The companies that innovate consistently promote people who generate the most valuable learning, even when the experiments fail. They ask: Who discovered an insight that changed our strategy? Who revealed a market dynamic we didn’t understand? Who ran experiments that taught us what to do next? That sends a signal: we value people who expand what we know, not just people who execute what we already know works. That’s the culture where breakthroughs happen.
Why This Matters More as You Scale
Early-stage startups have no choice but to experiment. Everything is unproven. Every customer is a learning opportunity. Failure is just Tuesday.
As you scale, something dangerous happens: you start to believe you’ve figured it out. You’ve found product-market fit. You’ve built a repeatable sales motion. You’ve identified your ICP. You’ve optimized your funnel. You shift from exploration mode to exploitation mode. From discovering what works to scaling what works. And for a while, it’s glorious. Growth is predictable. Efficiency improves. Investors are happy.
Then the market shifts. Your CAC creeps up. Your conversion rates drift down. Competitors copy your playbook. The thing that was working starts working less. And you realize: you’ve built a machine that’s incredible at optimizing the current game, but you’ve lost the muscle for discovering the next one. That’s the moment when companies plateau.
Not because they’re executing poorly. Because they stopped learning. They killed the experimental culture that got them to scale in the belief that scale required certainty. But scale doesn’t eliminate the need to experiment. It just changes what you’re experimenting with.
Early-stage experiments test: Can we find product-market fit?
Growth-stage experiments test: Can we find new channels, new segments, new use cases before our current ones saturate?
The companies that sustain growth through multiple S-curves are the ones that keep the experimental muscle strong even as they scale. They protect budget for experiments. They celebrate learning, not just wins. They hire people who are comfortable with uncertainty.
They understand that iteration only compounds if you keep iterating.
The Math of Compounding Learning
Here’s the thing most people miss about experimentation: It’s not about your hit rate. It’s about the rate at which your hits improve.
If you run ten experiments a quarter and one succeeds, that’s a 10% hit rate. Sounds terrible. But if each failed experiment teaches you something that makes the next experiment 5% more likely to succeed, your hit rate compounds.
Quarter one: 10% hit rate Quarter two: 15% hit rate (because you’re learning from failures) Quarter three: 23% hit rate Quarter four: 35% hit rate
By year two, you’re hitting on half your experiments because you’ve built pattern recognition. You’ve learned what signals to look for. You’ve developed intuition about what works in your market.
That’s what most organizations never get to. Because they see the 10% hit rate in quarter one and decide “experimentation doesn’t work here.” They never stay in the game long enough to let the learning compound.
How to Make Learning Compound
If you want experimentation to generate real value, you need three things:
- A knowledge system, not just a project tracker
Most teams track experiments in spreadsheets or project management tools. Launch date. Status. Results. But they don’t capture the why behind the results. The insights. The surprises. The patterns. You need a knowledge system: a central place where every experiment gets documented with:
- The hypothesis we were testing
- The execution approach we took
- The data we observed
- The insights we extracted
- The questions it raised for next experiments
- Links to related experiments
Make it searchable. Make it accessible. Make it required reading. Because the value isn’t in any single experiment. It’s in the pattern across experiments.
- Regular learning synthesis
Raw experiment data doesn’t turn into strategy on its own. Someone needs to synthesize it. I’ve seen this work best with quarterly learning reviews: the team steps back and asks:
- What patterns are we seeing across experiments?
- What have we learned about our market this quarter?
- What assumptions have been validated or invalidated?
- What should we do more of? Less of? Differently?
- What new hypotheses does this suggest?
That’s where individual experiments become institutional learning. That’s where iteration becomes compounding.
- Cross-functional learning loops
The marketing team learns something. The product team learns something. The sales team learns something. Customer success learns something. But those learnings stay siloed. They don’t cross-pollinate. The companies that learn fastest build explicit mechanisms for sharing:
- Monthly cross-functional experiment reviews
- Shared learning databases accessible to all teams
- Rotation programs where people spend time in other functions
- Rituals where insights flow bidirectionally
Because the real breakthroughs often come from connecting dots across functions. Marketing learns what message resonates. Product learns what feature creates retention. Sales learns what use case expands. Customer success learns what drives advocacy. Connect those insights, and you’ve got a thesis for how to grow. Keep them separate, and they’re just isolated data points.
The Bottom Line
I’ve watched companies spend millions optimizing things that were fundamentally broken. I’ve also watched companies stumble onto breakthroughs because someone tried something unscalable, learned something unexpected, and followed the insight. The difference is always the same: some teams treat experiments as gambles that need to win. Other teams treat them as investments in understanding.
One team asks: “Did it work?”
The other asks: “What did we learn?”
One team kills experiments that don’t scale.
The other uses them to discover what will.
Because here’s the truth nobody wants to say out loud: most of your growth experiments will fail. That’s not a bug. That’s how innovation works. You try ten things. Seven don’t work at all. Two work a little. One transforms your trajectory. But you can’t get to the one without running the seven. And you can’t make the seven valuable unless you extract the learning.
So the question isn’t whether you’re willing to fail. It’s whether you’re willing to learn.
Whether you can build a culture where experiments that don’t scale are celebrated for what they reveal, not dismissed for what they don’t achieve.
Whether you can document failures with the same rigor you document successes.
Whether you can let learning compound across time, across teams, across the organization.
Because that’s how you build the pattern recognition that makes the next experiment smarter. That’s how you develop the intuition that makes the next bet better.
That’s how iteration compounds. And in the long run, compounding iteration beats perfect execution every single time. Not because failure is good, but because learning is everything. And sometimes, if it doesn’t scale, it teaches you what will.
