Stop Experimenting. Start Shipping.

Photo by Andy Li on Unsplash

Steve Jobs said it best: “Real artists ship.”

The product management world has an experimentation obsession. Industry reports tell us only 31% of product teams prioritize rapid experimentation — and frame that as a crisis. But is it?

Here’s what the data actually says.

Experimentation Works — When It’s the Right Tool

Researchers from Harvard Business School and Duke University studied over 35,000 startups across several years. Teams that adopted A/B testing saw performance improve 30-100% in year one. They scaled faster. They also failed faster — which is its own kind of advantage.

But notice the finding: the gains came from adopting a discipline of testing, not from running the most tests. More experiments didn’t mean better outcomes. Focused, intentional testing did.

Experimentation isn’t a volume game. It’s a judgment call.

Speed Kills — In a Good Way

Bain studied over 1,000 companies across a decade. The finding was clear: high-performing companies made decisions faster, with less effort, and executed more frequently. Those companies returned 4x more to shareholders and grew profits 5.5x faster than their peers. Companies that deliberated for months watched competitors pass them.

Eisenhardt studied decision-making in fast-moving industries and found the same pattern — faster strategic decisions correlated with better firm performance. Her work also showed that fast decision makers used more information, not less. Speed didn’t mean recklessness. It meant preparation.

I’ve lived this. We lost a hosted site with no disaster recovery. No failover. No runway. We needed a functional website back up — one that let customers find open locations and place orders — within hours, not weeks. There was no time for A/B tests, no time to evaluate three CMS options, no room for a discovery sprint. We built it, posted it, and kept the business running. Then we watched the data, adjusted in real time, and iterated until the primary site was restored to full functionality.

That wasn’t a failure of process. That was product management.

The Real Experiment Happens After Launch

Here’s what the experimentation-first crowd misses: shipping is the experiment.

You learn more in the first two weeks after launch than in six months of pre-launch testing. Real users, real behavior, real stakes. The teams that win aren’t the ones who tested the most before launch. They’re the ones who paid attention after it.

But — and this matters — paying attention requires skill. AI has made it easier than ever to build prototypes. Tools let you validate concepts in days. That’s great for speed. It doesn’t solve the harder problem.

The fundamental work of gathering data, managing it, interpreting it, and making it actionable — that’s still a craft. Good product managers are building that muscle. The rest are running experiments to avoid making decisions.

The Real Question

The industry keeps asking: “Are you experimenting enough?”

Wrong question.

The right question is: “Do you know which decisions need data and which ones need conviction?” The best PMs I’ve worked with know the difference. They test when the stakes justify the delay. They ship when the clock says go.

And then they watch, learn, and adjust. That’s not anti-experimentation. That’s experimentation done by people who actually have to deliver.

Sources: Koning, Hasan & Chatterji, “Experimentation and Start-up Performance: Evidence from A/B Testing,” Management Science. Bain & Company, “Measuring Decision Effectiveness.” Eisenhardt, K.M., “Making Fast Strategic Decisions in High-Velocity Environments,” Academy of Management Journal, 1989.