post

B2B product management

Artificial Intelligence (AI) & Machine Learning (ML)

Customer Research

Hardware

Digital Transformation

Our Privacy Policy is changing on April 5, 2022 - for more information and a description of the updates, <a href="https://mtpdevwpe.wpengine.com/privacy/" style="color:#FFFFFF;text-decoration:underline;">see the policy here.</a>

Switching back the focus from ‘Build only’ to ‘Build-measure-learn’

Product career special part 2: Making a change

Build what matters with vision-led product management

Lessons in fearless leadership

Building a product organisation from scratch by Kevin Mansfield

Footer About

Footer Contribute

Footer Get Involved

Footer Links

Mind the Product

Product Management

The executive product leader persona

Test blog post

Being a problem solver by heart and product manager by trade, Thomas has experience discovering, validating and developing digital products. Employing a hypothesis-driven approach, he takes feature or product ideas from conceptualisation to realisation by applying lean &amp; user-centered methods. Currently he is a Product Manager at DAZN in Amsterdam, focused on Acquisition for the worlds leading OTT sports streaming provider. Prior he worked with German tech consultancy inovex in their internal incubator, after gaining experience as Product Manager at insurtech GetSafe and ticketing provider CTS Eventim. During the inaugural MTP Engage Hamburg he enjoyed the sense of community so much that he decided to co-found ProductTank Karlsruhe shortly afterwards – and to this day loves to bring the product community together to share and learn.

Thomas Leitermann

<p>Ready to get started with your experiment? Only need to find the right tool now? But which one to choose? It feels like being in a jungle with tools upon tools &#8211; how to navigate this situation?</p>
<p>In this talk from MTP Engage in Hamburg, I share learnings and examples from some of the numerous experiments I ran over the past two years at Inovex. I look at how we can learn from our mistakes, see how easy it is to get started with today’s tools and hear why I believe experimentation is more than A/B testing.</p>
<h2>Pack the Treasure Map and Compass</h2>
<p>Running experiments for experiments’ sake is a waste of time and money. We should always know WHY we want to run an experiment &#8211; what is our goal? How do we know we have achieved our goal? To structure the numerous assumptions and ideas and to think about what success looks like and how to measure it, at Inovex came to rely on the <a href="https://blog.strategyzer.com/posts/2015/3/5/validate-your-ideas-with-the-test-card" target="_blank" rel="noopener noreferrer">Test Card</a> from Strategyzer.</p>
<ul>
<li>It serves as the treasure map for our jungle expedition. On it, we gather all the important information to navigate the tool jungle. The metric defined on the Test Card serves us our compass, informing our decisions.</li>
</ul>
<h2>Pick the Right Machete</h2>
<p>When choosing the type of experiment, we have discovered two constraints. Is our hypothesis generative or evaluative? And, how strong does our evidence need to be, compared to the time we have?</p>
<p>When our hypothesis is generative it is either about the customer segment or the solution idea. If our hypothesis is evaluative it is either about revenue and willingness to pay, or acceptance of the solution. This positioning determines the experiment types at our disposal. We can then choose from a wide variety of experiments, for example. from the more than 30 experiment types described in the <a href="http://www.realstartupbook.com/" target="_blank" rel="noopener noreferrer">Real Startup Book</a>.</p>
<p>Once we find an experiment type that matches both our hypothesis and our desired strength of evidence, we need to find time and budget to be able to start.</p>
<ul>
<li>For a jungle expedition, this means we now can pack our machete, the experiment type that helps us to cut through the tool jungle. Time and budget can be thought of as food and water for our jungle expedition.</li>
</ul>
<h2>Ready to Go? &#8211; Almost</h2>
<p>Now we have all the necessities and are ready to go, right? Before going head over heels into the unknowns of the tool jungle I’d offer some knowledge acquired from previous expeditions.</p>
<p>In my experience experiments help to enforce the &#8211; hopefully old &#8211; mantra “you’re not the user” very well. Rather than build final product features based on assumptions, experiments help us expose those. With experiments we can learn from the people living in the jungle &#8211; our users.</p>
<p>In order to expose and learn from our users we first need to connect with them. Automating the recruitment of users for research and testing helps tremendously. There are a lot of useful tools available to find, acquire, schedule, and set up users for research. With those, we can put our user research recruitment on auto-pilot.</p>
<ul>
<li>In the jungle, we can navigate using maps and compass &#8211; but a GPS device gets us to the goal much quicker.</li>
</ul>
<h2>The Simpler, the Better &#8211; and Faster</h2>
<p>When looking for tools it&#8217;s easy to get down the rabbit hole to find the best, easiest, or latest tool. At Inovex we once spent a number of days rebuilding an experiment on a new tool after the first tool failed us in a small but vital part. So it’s best always to choose the simplest tool. Instead of a high-fidelity prototyping tool &#8211; why not use Keynote or PowerPoint? Simple tools also have the added benefit that everyone should know them well. So last-minute changes can be done quickly and without someone becoming a bottleneck.</p>
<ul>
<li>If you’re in the jungle looking to chop firewood then you could use a power saw. But an axe also gets the job done and never runs out of fuel.</li>
</ul>
<h2>User-Test your User Testing Tools</h2>
<p>There are fantastic tools available to help us understand user behaviour in our experiments, tools like Hotjar, Mixpanel, or Amplitude. They also help with measuring our metric, the compass of our experiments. To be able to rely on metrics we need to make sure to test the tools end-to-end. Only then are we sure they are capable of delivering the metric we need.<br />
At Inovex we learned this the hard way. We ran an experiment for two days before discovering the metric we needed could not be produced by the tool. Only after switching tools and re-running the experiment were we able to answer our experiment hypothesis.</p>
<ul>
<li>Before going on a multi-day jungle expedition, it&#8217;s probably better to survive one night. Make sure your user testing and measuring tools survive one user.</li>
</ul>
<h2>Iterate Within the Experiment</h2>
<p>Once the experiment is running there is nothing to do but wait, right? I’d argue differently. The probability of getting everything right the first time is negligible. I believe the well-known agile motto of inspect and adapt should also be used within an experiment. In one of our experiments at Inovex, we set up ads to drive traffic to the experiment throughout the night, then inspected the results and identified issues in the morning, applied changes in the afternoon and started the cycle again in the evening.</p>
<ul>
<li>Making fire with a flint rarely works the first time &#8211; but if we inspect and adapt we can get the fire for our overnight stay in the jungle going quickly.</li>
</ul>
<h2>Find a Path Through the Tool Jungle</h2>
<p>Equipped with learnings from previous expeditions as well as a backpack with treasure map, compass, machete and supplies I am confident there is a path through the tool jungle.</p>
<p><img decoding="async" src="https://mtpdevwpe.wpengine.com/wp-content/uploads/2019/08/e0521808-321c-424a-b49e-2936c447f46e_MTP-Engage_Thomas-Leitermann_EN.png" alt="" /></p>
<p>The slidedeck from this talk can be found <a href="https://speakerdeck.com/thomasleiterman/navigating-the-tool-jungle-how-to-pick-the-right-tools-for-your-hypothesis-experiments" target="_blank" rel="noopener noreferrer">here (English)</a> and <a href="https://speakerdeck.com/thomasleiterman/experimente-and-tools-wie-man-sich-in-diesem-dschungel-zurechtfindet" target="_blank" rel="noopener noreferrer">here (German)</a>.</p>
<div class="g g-21"><div class="g-single a-93"><a class="gofollow" data-track="OTMsMjEsMSw2MA==" href="https://mtpdevwpe.wpengine.com/product-management-training/public-workshops/communication-alignment-workshop/"><img decoding="async" src="https://mtpdevwpe.wpengine.com/wp-content/uploads/2022/04/Training-CA-layer-of-strategy-FOOTER-scaled.jpg" /></a></div></div>

core/freeform

Hypothesis and Testing

What’s stopping teams from running experiments by Cameron Savage

How to be a better data-driven product manager

Product’s role in experimentation culture by Terry Lee

Testing Business Ideas, by David Bland

Hypothesis Driven Validation By Nate Archer

How to Pick the Right Tools for your Experiments by Thomas Leitermann

Video: What you testing? Minimum Viable People, Product & Platform

Video: Easy A/B Testing F**k-Ups

Video: How to love A/B and Multivariate Testing

ProductTank Abu Dhabi

ProductTank Manchester

ProductTank Newcastle Upon Tyne

ProductTank Milton Keynes

ProductTank London

ProductTank Jacksonville

ProductTank Phoenix

ProductTank Ljubljana

ProductTank Belgrade

ProductTank Bologna

ProductTank Edinburgh

ProductTank München

ProductTank Birmingham

ProductTank Porto

ProductTank Pune

ProductTank Chennai

ProductTank Genève

ProductTank Rotterdam

ProductTank Heidelberg

ProductTank Málaga

ProductTank Paris

ProductTank Hamburg

ProductTank Kraków

ProductTank Riga

ProductTank Reading

ProductTank San Francisco

ProductTank Montréal

ProductTank Goiania

ProductTank Des Moines

ProductTank Bangkok

ProductTank Canberra

ProductTank Dubai

ProductTank Valencia

ProductTank Madrid

ProductTank Düsseldorf

ProductTank Norwich

ProductTank Bucharest

ProductTank Raleigh

ProductTank Warsaw

ProductTank Bielefeld

ProductTank Bern

ProductTank Tokyo

ProductTank Pittsburgh

ProductTank Cardiff

ProductTank Krakow

ProductTank NYC

Product Leadership Course

ProductTank Recife

ProductTank Dublin

Member AMA: Unpacking User Research with Laura Klein

ProductTank Copenhagen

Trainer-Led Workshops

ProductTank Tauranga

World Product Day

Member AMA with Ken Norton

#mtpcon London+EMEA 2022 Workshops

#mtpcon London+EMEA 2022 Conference