post

B2B product management

Artificial Intelligence (AI) & Machine Learning (ML)

Customer Research

Hardware

Digital Transformation

Our Privacy Policy is changing on April 5, 2022 - for more information and a description of the updates, <a href="https://mtpdevwpe.wpengine.com/privacy/" style="color:#FFFFFF;text-decoration:underline;">see the policy here.</a>

Switching back the focus from ‘Build only’ to ‘Build-measure-learn’

Product career special part 2: Making a change

Build what matters with vision-led product management

Lessons in fearless leadership

Building a product organisation from scratch by Kevin Mansfield

Footer About

Footer Contribute

Footer Get Involved

Footer Links

Mind the Product

Product Management

The executive product leader persona

Test blog post

Eira is an editor for Mind the Product. She's been a business journalist, editor, and copywriter for longer than she cares to think about.

Eira Hayward

<p>In this #mtpcon <span style="text-decoration: underline;"><a href="https://mtpdevwpe.wpengine.com/mtpcon/san-francisco/" target="_blank" rel="noopener">SF+Americas 2022</a></span> Spotlight session, sponsored by <span style="text-decoration: underline;"><a href="https://launchdarkly.com/" target="_blank" rel="noopener">LaunchDarkly</a></span>, the company’s Principal Product Manager Cameron Savage looks at what stops teams from running experiments.</p>
<p>Watch this video or read on for key highlights from the talk.</p>
<ul>
<li>Experiments can be seen as additional work</li>
<li>Commonly challenges come with engineering, data ingestion, decision making and product culture</li>
<li>Teams can find it hard to capture data and to work with it</li>
<li>Psychological safety and support are needed for successful experiments</li>
</ul>
<p>Cameron heads up the decision science team at feature management platform LaunchDarkly and talks to hundreds of customers of all shapes and sizes in different industries and with different goals.  He says these conversations show a fairly consistent set of themes that prevent product teams from measuring their work in a way that they feel good about. In this talk he looks at the friction areas and challenge points in experimentation and at what you can do to get your team to start running experiments or to run more of them.</p>
<h2>What is an experiment?</h2>
<p>An experiment, says Cameron, is a way to compare an old version against a new version or many versions against themselves so that you quantify the impact of decisions you make and increase certainty in your decision making. He says: “One of the superpowers of experimentation is that you isolate exclamation points and question marks such that you can understand the relative impact of a change on your system regardless of all the other changes in your system.” A product manager’s goal is to identify the right decisions to make, he adds, and experiments remove subjective noise to give you a higher level of certainty.</p>
<h2>Why aren’t more product teams running experiments?</h2>
<p>Cameron has found there are four friction areas that prevent teams from running experiments or running as many as they would like. They are:-</p>
<p><span style="font-weight: bold;">Engineering</span>: issues around workflow, speed, risk. Teams see experiments as additional work to implement when they need to focus on getting the things they’ve already scoped out the door. It adds to complexity and delivery time and can add to security and performance risks.</p>
<p>Cameron recommends that by placing equal priority on tooling that fits the dev teams’ workflows you can run more experiments in the long run.</p>
<p><span style="font-weight: bold;">Data ingestion</span>: issues around effort, reliability, trust. Teams may feel they have a hard time connecting the data they need and trusting that is reliable enough to use to make decisions. Setting up data intake is real work, says Cameron, and it needs to be properly planned. There may be challenges in ensuring data is captured in a scalable way and in validating that data is accurate. Cameron recommends that you partner with data science/ engineering upfront to map out data flows and it should mean that going forward you get less pushback about adding new data points or locations.</p>
<p><span style="font-weight: bold;">Decision making</span>: issues around comprehension and trust. Teams aren’t confident about running their own experiments. This lack of confidence may start with concern about how to interpret statistics, says Cameron, a lack of trust that teams know how to set up experiments, or that there is enough data to get results. To overcome these challenges Cameron recommends bringing delivery teams into evaluation and planning of tooling in order to assess how the results will be used in practice. You can then better democratise data-driven decisions and free up your experts to focus on more strategic efforts.</p>
<p><span style="font-weight: bold;">Product culture:</span> issues around momentum, fear, buy-in. Cameron says teams can be so focused on speed of delivery that they are afraid that experiments will slow them down. Often there’s a lack of priority, comfort or habit around using data to make decisions, teams quantify their success by looking at how often they launch things and failed experiments equate with failure. “If you’re unwilling to take the time to test and validate something that you’re launching then essentially you’re in a feature factory, “ Cameron says. It’s a challenge for leadership, he adds, and recommends that teams are given safety and support so that they can take time to validate releases.</p>
<h2>There&#8217;s more where that came from!</h2>
<p>Explore more <a href="https://mtpdevwpe.wpengine.com/mtpcon-sf-video-library-2022/" target="_blank" rel="noopener">#mtpcon SF+Americas 2022 video content</a> or use our <span style="text-decoration: underline;"><a href="https://mtpdevwpe.wpengine.com/content-a-z/" target="_blank" rel="noopener">Content A-Z</a></span> to find even more product management insights.</p>
<div class="g g-21"><div class="g-single a-93"><a class="gofollow" data-track="OTMsMjEsMSw2MA==" href="https://mtpdevwpe.wpengine.com/product-management-training/public-workshops/communication-alignment-workshop/"><img decoding="async" src="https://mtpdevwpe.wpengine.com/wp-content/uploads/2022/04/Training-CA-layer-of-strategy-FOOTER-scaled.jpg" /></a></div></div>

core/freeform

Hypothesis and Testing

What’s stopping teams from running experiments by Cameron Savage

How to be a better data-driven product manager

Product’s role in experimentation culture by Terry Lee

Testing Business Ideas, by David Bland

Hypothesis Driven Validation By Nate Archer

How to Pick the Right Tools for your Experiments by Thomas Leitermann

Video: What you testing? Minimum Viable People, Product & Platform

Video: Easy A/B Testing F**k-Ups

Video: How to love A/B and Multivariate Testing

ProductTank Krakow

ProductTank Heidelberg

ProductTank NYC

Product Leadership Course

ProductTank Recife

ProductTank Dublin

ProductTank London

ProductTank Birmingham

Member AMA: Unpacking User Research with Laura Klein

ProductTank Reading

ProductTank Copenhagen

Trainer-Led Workshops

ProductTank Tauranga

World Product Day

Member AMA with Ken Norton

ProductTank Montevideo

ProductTank Tallinn

ProductTank Edinburgh

ProductTank Abu Dhabi

ProductTank Oslo

ProductTank Vilnius

ProductTank Brussels

ProductTank Bristol

ProductTank Norwich

ProductTank Brno

ProductTank Cardiff

ProductTank Newcastle Upon Tyne

ProductTank Barcelona

ProductTank Bucharest

ProductTank Pittsburgh

ProductTank Atlanta

ProductTank Los Angeles

ProductTank Palo Alto

ProductTank Pune

ProductTank Tel Aviv-Yafo

ProductTank Tokyo

ProductTank Dubai

ProductTank Madison

ProductTank Chicago

ProductTank Brisbane

ProductTank Bonn

ProductTank Des Moines

#mtpcon London+EMEA 2022 Workshops

#mtpcon London+EMEA 2022 Conference

ProductTank San Francisco

ProductTank Washington

ProductTank Orlando

ProductTank Köln