post

B2B product management

Artificial Intelligence (AI) & Machine Learning (ML)

Customer Research

Hardware

Digital Transformation

Our Privacy Policy is changing on April 5, 2022 - for more information and a description of the updates, <a href="https://mtpdevwpe.wpengine.com/privacy/" style="color:#FFFFFF;text-decoration:underline;">see the policy here.</a>

Switching back the focus from ‘Build only’ to ‘Build-measure-learn’

Product career special part 2: Making a change

Build what matters with vision-led product management

Lessons in fearless leadership

Building a product organisation from scratch by Kevin Mansfield

Footer About

Footer Contribute

Footer Get Involved

Footer Links

Mind the Product

Product Management

The executive product leader persona

Test blog post

Eira is an editor for Mind the Product. She's been a business journalist, editor, and copywriter for longer than she cares to think about.

Eira Hayward

<p>Experiments are a quick and inexpensive way for product managers to validate and prioritise their ideas. In <a href="https://mtpdevwpe.wpengine.com/experimentation-for-product-teams-part-1/" target="_blank" rel="noopener">Experimentation for product teams: Part 1</a> we explore the types of experimentation product managers can use and at some of the pitfalls and challenges.</p>
<p>But, to really understand experimentation, we want war stories! So, in this second part, we take a look at some real-life experimentation experiences as members of the product community share their tales – including what went right and what went wrong.</p>
<h2>Sports streaming user sign-up</h2>
<p><a href="https://www.linkedin.com/in/terry-lee-296a089/?originalSubdomain=uk" target="_blank" rel="noopener">Terry Lee</a> is an acquisition and onboarding specialist, currently Senior Product Manager at <a href="https://flo.health/" target="_blank" rel="noopener">Flo Health</a>, and his example comes from one of his previous roles at sports streaming platform, <a href="https://www.dazn.com/en-GB/welcome/" target="_blank" rel="noopener">Dazn</a>.</p>
<p>He says: “The hypothesis was good, the tests were good, but the vision was so short-term that the impact was negative.</p>
<p>“We were doing a three-step onboarding process, a landing page, then sign-up page, and then the payment page. The landing page was designed to give the user information about the product. The sign-up page was pretty blank because we didn&#8217;t want to distract users from the information they needed to enter. And then the payment page was also very blank.</p>
<p>“Our short-term goal was to improve the numbers of users going from the landing page to the sign-up page, only one step. Our hypothesis was that ‘if we change the CTA button to yellow we will improve the numbers of users going from the landing page to the sign-up page’. And we did. We saw something like a 60% improvement in users going from the landing page to the next screen. Fantastic, we all patted ourselves on the back, we’d achieved our goal. We also got a good improvement on signups which was also great.</p>
<p>“But we got a massive reduction in the performance of our payment page. Why had changing one button on the front end changed the performance of a page to two pages later? We talked to a few of our users and found that the landing page had been where users understood what the product was, what platforms it worked on and so on. All that information on the landing page was being skipped, the sign-up page didn’t present a problem because it was just an email, but when users got to the payment page they’d ask themselves, ‘what am I paying for, what device does this work on?’.”</p>
<p>All of that initial information had stayed on the landing page, but by putting a big yellow Next button on the page Terry and his team were essentially pushing users on quickly to the next page so that they didn’t take it in.</p>
<h2>Complex food delivery AI</h2>
<p><a href="https://uk.linkedin.com/in/graham-paterson-938b861b" target="_blank" rel="noopener">Graham Paterson</a> is a long-time product leader. Currently Product Partner at seed stage venture capital firm <a href="https://www.connectventures.co/" target="_blank" rel="noopener">Connect Ventures</a>, he’s spent time managing product at Deliveroo and at Wise, among other companies. Says Graham: “What makes me nervous about experimentation is that there are lots of things that can go wrong.” He says that it’s easy to set an experiment up incorrectly so that it doesn’t address the question you’re trying to answer. He also cautions against what he calls experimentation theatre – where you want to show the CEO that you’re doing something and you run experiments that don’t really move your understanding forward. “You should always talk to your users and analyse what people actually do,” he says.</p>
<p>His experiences of building an AI system to work out which order to place with which rider at food delivery business <a href="https://deliveroo.co.uk/?utm_source=google&amp;utm_medium=cpc&amp;utm_term=deliveroo&amp;utm_campaign=%2A%2A%5EAcquisition%5ESearch%5EBrand%5EUK%5ELondon%5E%5EExact%5EAPI%5E%5E%5E%5E%5EEN%5ELifecycle%5E%C2%A315841156077&amp;utm_loc=1007177&amp;utm_device=c&amp;utm_adposition=&amp;utm_network=g&amp;utm_targetid=kwd-52577441545&amp;gclid=CjwKCAjwrfCRBhAXEiwAnkmKmRCDttg0D6fN2jGn5sq_BfqJFgFn9ihQf1hF6p3pQ7PI458XMc3tUhoCLHoQAvD_BwE&amp;gclsrc=aw.ds" target="_blank" rel="noopener">Deliveroo</a> highlight how complex experimentation can become. “It surprised me how important it was to test each part of the system we built,” he says. When testing with users – whether they were people ordering the food or Deliveroo riders – it was very hard to predict what they would do. “People are unique and emotional,” he adds.</p>
<p>It was hard to know whether an algorithm would definitely improve things because it might cause something strange to happen somewhere else in the system. “For example, if we were to allocate riders and do it too well that would mean the restaurants won’t prepare the food on time and the riders would be left waiting. So even if the algorithm theoretically chooses better riders, because of the restaurant lag that improvement doesn’t work out.”</p>
<p>“We made another improvement, one that was incredibly powerful, and we thought it would improve the system efficiency by 50%. But we found it was pretty much only picking scooter riders rather than cyclists.</p>
<p>“We weren’t just building for London, we were building for a whole bunch of cities around the world. Sydney harbour cuts the city in half, there are lots of hills in Hong Kong, all these types of factors played a part. We had to make sure our improvements made sense everywhere rather than just in this mathematically perfect place we modelled. There were also other factors that could skew the results of our experiments – maybe it&#8217;s a Saturday night, in which case there are more orders and more riders and so the whole system is more efficient, but that would skew the results.”</p>
<h2>Keep talking to customers</h2>
<p>An experiment that shows the importance of continuing to talk to customers comes from Graham’s time at online money transfer business <a href="https://wise.com/" target="_blank" rel="noopener">Wise</a> (then TransferWise). One of his colleagues decided to run an experiment to move the user sign-in point to much earlier in the flow.</p>
<p>Says Graham: “I couldn’t understand why they did this. All the UX design rules state you should ask people for commitment as late as possible. But he tested moving it right to the front of the flow, because he’d had some insight from some customers that this might be a good thing to do.</p>
<p>“To my surprise it drastically improved conversion. Because people were inputting quite sensitive information, they felt a lot more comfortable doing it behind a password. It showed me that you may think your intuition is right most of the time but it can definitely be wrong.”</p>
<p>Graham concludes with some advice about how important it is to have a holistic view of your product. “People will pick a metric they want to test and prove it’s the one. They might get 50% more customers clicking through a button and decide their experiment is a resounding success. But if they don’t look at the real measure of success, which is how many people actually bought the product in the end, then they’ll have no idea of the impact they’re having on the business.&#8221;</p>
<h2>Share your insights</h2>
<p>There&#8217;s nothing better than hearing real-world examples from other product managers and for that reason, we are always looking for case studies from our community.</p>
<p>To help us to share best practice and advice on all aspects of product management, all you need to do is speak to us, we&#8217;ll do the rest!</p>
<p>If you&#8217;re interested in sharing your case study, simply let us know. <a href="https://mindtheproduct.typeform.com/to/oq6jksWh?typeform" target="_blank" rel="noopener">Complete our quick form</a></p>
<h2>Further experimentation reading</h2>
<ul>
<li><a href="https://mtpdevwpe.wpengine.com/products-role-in-experimentation-culture-by-terry-lee/" target="_blank" rel="noopener">Product’s role in experimentation culture by Terry Lee</a></li>
<li><a href="https://www.youtube.com/watch?v=MOlVV7CjtB8" target="_blank" rel="noopener">Stay humble or be humbled — Case studies from the experimentation trenches by Anthony Rindone</a></li>
<li><a href="https://mtpdevwpe.wpengine.com/using-experimentation-drive-product/" target="_blank" rel="noopener">Using experimentation to drive product – Stephen Pavlovich (CEO of Conversion.com)</a></li>
</ul>
<div class="g g-21"><div class="g-single a-95"><a class="gofollow" data-track="OTUsMjEsMSw2MA==" href="https://mtpdevwpe.wpengine.com/product-management-training/public-workshops/product-metrics-workshop/"><img decoding="async" src="https://mtpdevwpe.wpengine.com/wp-content/uploads/2022/04/Training-Metrics-what-should-a-PM-know_-FOOTER-scaled.jpg" /></a></div></div>

core/freeform

Product Management Skills

Don’t convince. Inspire! – Lessons on storytelling by Petra Wille

Strengthening collaboration between business and product

Taming the roadmap circus by Maarten Dalmijn

Design for cognitive Bias by David Dylan Thomas

The role of the first product manager by Andrew Moll

It’s all about the outcome by Georgie Smallwood

Transition from a Feature Factory to Business Impact team by Sandra Hinz

Sharpening your product craft as an internal product manager by Alison Hickson

From burnout to bounceback by Sara Wachter-Boettcher

The importance of EQ for product development teams by Nikkia Reveillac

Experimentation for product teams: Part 2

ProductTank Abu Dhabi

ProductTank Manchester

ProductTank Newcastle Upon Tyne

ProductTank Milton Keynes

ProductTank London

ProductTank Jacksonville

ProductTank Phoenix

ProductTank Ljubljana

ProductTank Belgrade

ProductTank Bologna

ProductTank Edinburgh

ProductTank München

ProductTank Birmingham

ProductTank Porto

ProductTank Pune

ProductTank Chennai

ProductTank Genève

ProductTank Heidelberg

ProductTank Málaga

ProductTank Paris

ProductTank Hamburg

ProductTank Kraków

ProductTank Riga

ProductTank Reading

ProductTank San Francisco

ProductTank Montréal

ProductTank Goiania

ProductTank Des Moines

ProductTank Bangkok

ProductTank Canberra

ProductTank Dubai

ProductTank Valencia

ProductTank Madrid

ProductTank Düsseldorf

ProductTank Bucharest

ProductTank Raleigh

ProductTank Warsaw

ProductTank Bielefeld

ProductTank Bern

ProductTank Tokyo

ProductTank Pittsburgh

ProductTank Cardiff

ProductTank Krakow

ProductTank NYC

Product Leadership Course

ProductTank Recife

ProductTank Dublin

Member AMA: Unpacking User Research with Laura Klein

ProductTank Copenhagen

Trainer-Led Workshops

ProductTank Tauranga

World Product Day

Member AMA with Ken Norton

#mtpcon London+EMEA 2022 Workshops

#mtpcon London+EMEA 2022 Conference

Experimentation for product teams: Part 2

Eira Hayward

Comments

Read next