The following insights were gleaned from a presentation at INDUSTRY given by Jon Noronha of Optimizely @thatsjonsense. Download Product Management TACTICS eBook for more.
When it comes to product development, experimenting badly can be worse than not experimenting at all. As one of the most powerful tools available to product teams, when done right, experimentation allows teams to iterate gradually and validate ideas with data.
We’re all familiar with the process of building and launching products. The problem is that until we know how it went, we’re essentially blind. There’s a better approach: Experiment and iterate from start to finish, repeatedly.
Product experimentation involves learning, building, measuring, and repeating. Once this happens, we can start to quantify the impact that product changes has. Failure will happen, and that’s OK — we should embrace it. In fact, failing can help us find bigger and better successes as long as we’re working to learn from those failures. But failure often happens from simple mistakes that can be avoided. Some of those mistakes include:
- Optimizing the wrong metrics. Most product people understand the importance of setting goals in place — but what happens if we’re setting the wrong goal? Put yourself in the user’s shoes. Ask, “What if this went up and nothing else?” Constantly re-evaluate and don’t underestimate the value of trusting your gut at times.
- Getting tricked by the stats. Don’t peek at A/B results too soon. There’s a risk of false positives — more than you’d normally see. If you’re in doubt, get a data scientist or statistician, or use a platform that can help you dissect the results at the right time.
- Thinking too small. Experimentation is often misunderstood as simply A/B testing… with a literal “A” and a “B.”. A better system would be A-B-C-D-E-F testing because at that level, creativity increases. Testing five or more variations can improve your win rate by 75%.
And like products, your experimentation process should always be open to iteration and improvement.
To view Jon’s full presentation, visit:
