post

B2B product management

Artificial Intelligence (AI) & Machine Learning (ML)

Customer Research

Hardware

Digital Transformation

Our Privacy Policy is changing on April 5, 2022 - for more information and a description of the updates, <a href="https://mtpdevwpe.wpengine.com/privacy/" style="color:#FFFFFF;text-decoration:underline;">see the policy here.</a>

Switching back the focus from ‘Build only’ to ‘Build-measure-learn’

Product career special part 2: Making a change

Build what matters with vision-led product management

Lessons in fearless leadership

Building a product organisation from scratch by Kevin Mansfield

Footer About

Footer Contribute

Footer Get Involved

Footer Links

Mind the Product

Product Management

The executive product leader persona

Test blog post

Maahir is a product leader with over a decade of experience building 0 to 1 products across both consumer tech and enterprise SaaS.

Maahir Shah

<p>Data is a critical cornerstone of informing every single product decision we make today. However, this means that there is room for error with significant adverse consequences if we leverage data incorrectly. Let’s look at all the wrong ways to be data-driven and how we can change that with a few concrete examples below.</p>
<p>You are leading a high priority area within your organization. The product your team is building and the work your team is doing is expected to directly impact and influence the Daily Active User (or other) northstar metric for the organization. Your team sets a goal to move the Daily Active User metric from X at the start of the year to Y at the end of the year. Your team builds and tests different product ideas to validate/invalidate your hypotheses.</p>
<p>Some of these ideas move northstar metrics and are shipped while others do not and are deprecated. The team ends the year exceeding the goal by 5%. Your team is expected to get three more engineers to continue the work going into the next year &#8211; that’s a 30% increase in capacity. You’re on track for your promotion. Things are looking great…</p>
<h2>Not so fast…here’s another view of the story above.</h2>
<p>Your team picked the metric before defining who your users are, understanding what they need and determining what value you wanted to deliver to them. As a result, you either ended up building some generic optimizations which grew Daily Active Users.</p>
<p>Your team had a bunch of ideas that could improve the overall product-market fit of your product. But, you were looking for bronze coins in a diamond mine. As a result, your team decided to de-prioritize product ideas which would solve real problems for your users because they wouldn’t immediately move northstar metrics.</p>
<p>A few ideas your team had, drove real value to end-users. But, the success and failure of these product ideas were evaluated by A/B tests and judged for their merit solely based on the impact on northstar metrics. As a result, your team decided to deprecate all of these ideas.</p>
<p>And while you’re on track for your promotion, you are having a harder and harder time shutting out the voice in your head that your job is no longer fulfilling and it might be time for your to move on.</p>
<p>Let’s break down how to use data correctly and incorrectly across the following three key aspects of the product development process:</p>
<ul>
<li>How we prioritize</li>
<li>What we measure</li>
<li>How we measure</li>
</ul>
<h2>How we prioritize</h2>
<p>As Product Managers, our goal is to improve the value that a product delivers for our end users &#8211; our goal is not to move a metric. The most common mistake a Product Manager can make for the long term success of a product is to let the metric define the product versus letting the user needs define the product and the product define the metric.</p>
<p style="text-align: center;"><span style="text-decoration: underline;"><a href="https://mtpdevwpe.wpengine.com/metrics-that-matter-live-panel/" target="_blank" rel="noopener">Discover more insights on creating metrics that matter.</a></span></p>
<p>How often have you made a prioritization decision based on the impact on a northstar metric? It’s rare and difficult for value-added product features to lead to big gains in the northstar metrics in a short period of time.</p>
<p>For instance, let’s take an example of an app that helps people discover local businesses. If a product manager decided that success was to grow Daily Active Users it’s easy for a new notification to lead to immediate and short term gains in Daily Active Users. However, if all this product manager did is optimize notification systems, over a longer period of time, we haven’t moved the needle on the value the product delivers to end-users at all and the efficacy of these notifications would decrease. Instead, this same product manager could take an alternate approach where success was to help users find the best plumbers for them in addition to restaurants.</p>
<p>The goal is not to discount the use of data. Data would be essential in making a good prioritization decision. But the right data questions to ask would not be by how much the feature could improve Daily Active Users.</p>
<p>The right questions instead are:</p>
<ul>
<li>How many plumbers do we have on the platform?</li>
<li>How many searches do we have for plumbers on the platform?</li>
<li>How long / how many outreaches does it take for a user to find a good plumber?</li>
</ul>
<p>A reason why this problem arises in the first place is organizational culture. It’s easy to justify your prioritization rationale based on impact to northstar metrics — especially with leadership teams who are well versed with these metrics. A solution to this problem is to always create a goal map and drive alignment on that goal map. A goal map in this example could look something like:</p>
<figure style="width: 423px" class="wp-caption aligncenter"><img loading="lazy" decoding="async" src="https://mtpdevwpe.wpengine.com/wp-content/uploads/2022/02/e620ff6d-0f11-4f3a-b5b3-dffe1f0c1ad9_Screen20Shot202022-01-1620at203.49.2420PM.png" alt="" width="423" height="281" /><figcaption class="wp-caption-text">An example of a goal map</figcaption></figure>
<p style="text-align: center;"><span style="text-decoration: underline;"><a href="https://mtpdevwpe.wpengine.com/prioritisation-for-product-managers-are-we-doing-it-right/" target="_blank" rel="noopener">Read more about prioritisation for product managers</a></span></p>
<h2>What we measure</h2>
<p>Given the aforementioned familiarity with and ease of communicating northstar metrics, we often tend to measure the success of a product based on how it impacts northstar metrics. The most common mistake a product manager can make is to put the cart before the horse.</p>
<p>When working on a new product, there are a few key ingredients to get right before we can start focusing on northstar metrics:</p>
<ul>
<li>Quality: Do we see a high number of crashes? Do pages take over 10 seconds to load? Is the product easy to use?</li>
<li>Value: Are we delivering value to our end users?</li>
<li>Product Market Fit: Are we delivering that value to a significant number of people consistently i.e. are adoption and retention high among our target audience?</li>
</ul>
<p>Again, data is a vital part of the entire journey of figuring out whether a product is successful or not. But, using the right data before we jump to northstar metrics is an important step we often tend to omit.</p>
<p>Organizational culture is difficult to combat here again if the entire organization is speaking the language of north star metrics. A solution to the problem is to proactively land a testing and launch strategy with clear phases (e.g. Phase 1A: Quality, Phase 1B, Usability, etc), timelines for each phase and exit criteria to move from one phase to another.</p>
<p style="text-align: center;"><a href="https://mtpdevwpe.wpengine.com/data-driven-product-management-by-matt-lemay/" target="_blank" rel="noopener">Read Data-Driven Product Management by Matt LeMay</a></p>
<h2>How we measure</h2>
<p>This one might be controversial but I would strongly recommend against using A/B tests that look at northstar metrics to make decisions on moving between phases of testing or making launch / deprecate decisions) for a few reasons:</p>
<ul>
<li>They typically don’t help us get a good signal on key metrics like Quality since your control group does not have access to the product feature at all.</li>
</ul>
<ul>
<li>They generally take very long to provide a statistically significant signal on metrics like retention.</li>
<li>They are rarely clean with all the different ML-based products running in our products.</li>
</ul>
<p>An alternate approach could be to use the phase-based approach described above with clearly defined benchmarks of exit criteria. For instance, with every product you work on, define benchmarks for clear success, signs of success and clear failure in each phase.</p>
<h2>In summary</h2>
<p>Building product continues to be a combination of art and science. Data is important in each step of the product development process. But to harness the power of data to yield long term success of your product, remember that:</p>
<ul>
<li>The value you deliver to end-users should define the product and hence the metric &#8211; the metric should not define the product.</li>
<li>Pick the right set of metrics to first understand whether you are delivering value to end-users before focusing on northstar metrics.</li>
<li>Leverage goal maps to showcase how your product will ladder up to northstar metrics over time.</li>
</ul>
<p style="text-align: center;"><span style="text-decoration: underline;"><a href="https://mtpdevwpe.wpengine.com/category/data-driven-product-management/" target="_blank" rel="noopener">Discover more insights on Data-Driven Product Management</a></span></p>
<div class="g g-21"><div class="g-single a-93"><a class="gofollow" data-track="OTMsMjEsMSw2MA==" href="https://mtpdevwpe.wpengine.com/product-management-training/public-workshops/communication-alignment-workshop/"><img decoding="async" src="https://mtpdevwpe.wpengine.com/wp-content/uploads/2022/04/Training-CA-layer-of-strategy-FOOTER-scaled.jpg" /></a></div></div>

core/freeform

Guest Post

Building Consumer-Grade Experiences for the Enterprise By Ciara Peter

What Makes a Great Product Manager by Graham Siener

The GIST Framework by Itamar Gilad

Scaling the Product and the Team by Rachael Shah

Decisions, Debt, and Other Dilemmas by Janna Bastow

Hacking Your Product Career by Gibson Biddle

How Medicine and Research Translates to Patients by Shree Patel

The Future of Product Management by Richard Harrison

Data Science and Digital Businesses by Ali Gulez

You Need A Product Strategy by Sam Odio

The wrong ways to be data driven

ProductTank Baku

ProductTank Rotterdam

ProductTank London

ProductTank Portland

ProductTank Thanh Pho Ho Chi Minh

ProductTank Stockholm

ProductTank Amsterdam

ProductTank Pittsburgh

ProductTank Auckland

ProductTank Brno

ProductTank Cardiff

ProductTank Montréal

ProductTank Waterloo

ProductTank Valletta

ProductTank Vilnius

ProductTank Belfast

ProductTank Curitiba

ProductTank Lausanne

ProductTank Malmö

ProductTank München

ProductTank Oxford

ProductTank Barcelona

ProductTank Seattle

ProductTank Madison

ProductTank Köln

ProductTank Norwich

ProductTank Bangkok

ProductTank Heidelberg

ProductTank Manchester

ProductTank Newcastle Upon Tyne

ProductTank Philadelphia

ProductTank Chicago

ProductTank Reston

ProductTank Edinburgh

ProductTank Valencia

ProductTank Paris

ProductTank Exeter

ProductTank Birmingham

ProductTank Porto

ProductTank San Francisco

ProductTank New York

ProductTank Orlando

ProductTank Tokyo

ProductTank Madrid

ProductTank Leeds

ProductTank Zürich

ProductTank Perth

ProductTank Los Angeles

ProductTank Des Moines

ProductTank Bucharest

ProductTank Bielefeld

ProductTank Krakow

ProductTank NYC

Product Leadership Course

ProductTank Recife

ProductTank Dublin

Member AMA: Unpacking User Research with Laura Klein

ProductTank Reading

ProductTank Copenhagen

Trainer-Led Workshops

ProductTank Tauranga

World Product Day

Member AMA with Ken Norton

#mtpcon London+EMEA 2022 Workshops

#mtpcon London+EMEA 2022 Conference

The wrong ways to be data driven

Maahir Shah

Comments

Read next