post

B2B product management

Artificial Intelligence (AI) & Machine Learning (ML)

Customer Research

Hardware

Digital Transformation

Our Privacy Policy is changing on April 5, 2022 - for more information and a description of the updates, <a href="https://mtpdevwpe.wpengine.com/privacy/" style="color:#FFFFFF;text-decoration:underline;">see the policy here.</a>

Switching back the focus from ‘Build only’ to ‘Build-measure-learn’

Product career special part 2: Making a change

Build what matters with vision-led product management

Lessons in fearless leadership

Building a product organisation from scratch by Kevin Mansfield

Footer About

Footer Contribute

Footer Get Involved

Footer Links

Mind the Product

Product Management

The executive product leader persona

Test blog post

Jase Clamp is a product consultant on a mission to see an Australian company in the top five global market cap companies. He believes the key to this is good product management. He serves on the Mind The Product team as Asia-Pacific Co-ordinator for ProductTank as well as consulting with startups on their product strategy and process.

Jase Clamp

<p>There were a lot of sessions at our recent <a href="https://mtpdevwpe.wpengine.com/mtpcon/london/" target="_blank" rel="noopener noreferrer">MTPCon</a> APAC on Discovery. Really nothing represents the core of our craft more than connecting with customers and building empathy for their struggles. Following a round table, I thought I’d take some time to write down some recent tips/tricks that I’ve used for scaling discovery.</p>
<h2>Lucky for us, research ain’t easy</h2>
<p>In product land, we all hear the cliched advice ‘talk to your customers’ as if you can just talk to them and ask them ‘what do you want us to build?’ and they’ll tell us the right answers.</p>
<p>Fortunately what keeps us product people gainfully employed (but not so fortunate for innovation and progress in general) is that it’s not that simple.</p>
<p>Luckily there’s a method to our madness. We have at our disposal a great spectrum of user research methods. <a href="https://www.linkedin.com/in/crohrer/" target="_blank" rel="noopener noreferrer">Christian Rohrer</a> has a great map of research methods (below) so as a reference, we’re exploring the bottom left—discovery interviews.</p>
<figure style="width: 1194px" class="wp-caption alignnone"><img loading="lazy" decoding="async" src="https://mtpdevwpe.wpengine.com/wp-content/uploads/2021/05/ec4e97c7-6143-4369-a720-a4d965b7a67f_pasted20image200.png" alt="" width="1194" height="939" /><figcaption class="wp-caption-text">A landscape of user research methods (Image: Christian Rohrer)</figcaption></figure>
<h2>Discovery interviews</h2>
<p><a href="https://mtpdevwpe.wpengine.com/best-practices-for-designing-products-that-are-desirable-viable-and-feasible/" target="_blank" rel="noopener noreferrer">Discovery interviews</a> can be extra helpful when your &#8220;next&#8221; customer segments are different to your &#8220;now&#8221; segments. Also adopting new problem space or partner platforms can be a good prompt to do fresh discovery research. I recently faced a situation where 80-90% of our research candidates needed to be found outside of our existing user base. If you&#8217;re working on a mature product then you&#8217;d be spending the majority of your time interviewing existing customers to improve retention.</p>
<p>It&#8217;s helpful to plan your research ops approach by considering the phases of the research pipeline:</p>
<p>1. Campaign planning (hypothesis, target profiles, recruitment mechanisms)</p>
<p>2. Recruitment (message crafting, searching, cold reach out)</p>
<p>3. Booking (calendar and appointment mechanisms)</p>
<p>4. The interview (principles around running them)</p>
<p>5. Analysis and follow up (note storage, candidate follow up)</p>
<p>I’ll explore these below.</p>
<h2>Campaign planning</h2>
<p>I’ll start a campaign by establishing as clear a profile as possible for who we want to talk to, what our assumptions and hypotheses are about them, and listing out example questions that may be used in interviews.</p>
<p>I’ve used confluence to set up a master page on the campaign because we&#8217;d end up attaching notes from each interview underneath the master page with various data points rolled up. I’ve since also found tools like Dovetail which are more purpose-built for storing research findings.</p>
<p>Our target profile would include mainly behaviours target candidates are exhibiting—for example the types of jobs they&#8217;re trying to do—followed by potential role titles they might hold.</p>
<p>We’d also brainstorm various ways we might be able to find these people including user database filters, app surveys, emails, LinkedIn searches, and other various lead sources.</p>
<h2>Recruitment</h2>
<p>I started out doing this job myself but realised it takes a lot of time and a skills set you’d find in business development or human resource recruitment. In pursuing this I used a LinkedIn Recruiter for message credits and for reaching out to people.</p>
<p>Good message crafting became an important element impacting conversion rate to interviews. For each campaign, I iterated at least a dozen different message versions for reaching out. What I learned (which ironically may be common sense to those working in sales/recruitment):</p>
<ul>
<li>Sometimes short is not always best &#8211; for those who reach the end of the message, their sunk cost in reading may influence them to engage</li>
<li>Personal connections or references are number one &#8211; even if it’s the fact that they or you commented on a shared post etc, it helps</li>
<li>Make the purpose extremely clear upfront &#8211; this is research, we’re not trying to sell you anything, you’re helping us</li>
<li>We found removing the word “interview” and replacing it with “research chat” helped</li>
<li>Make the call to action clear &#8211; e.g. a booking/calendar link</li>
<li>Incentives do help &#8211; we called them “small tokens of appreciation for their time” &#8211; and offer where the target candidate is of a high perceived value</li>
<li>Share your mission to change the world and how pivotal they are in helping advance that</li>
</ul>
<h3>Trying self-service research platforms</h3>
<p><span style="font-weight: 400;">Recruitment itself was probably the biggest problem I had to figure out in our research ops. Being a product person, I also tried research platforms such as respondent.io, userresearch.com and askable.com. The way these platforms work is they have varying pool sizes of candidates (e.g. 100-500k) which they stock up via ads and other mechanisms from time to time. You can filter your research request demographically to certain profiles. </span></p>
<p><span style="font-weight: 400;">The revenue model these platforms generally operate on is a referral fee which might be a percentage of the incentive offered. In using these platforms I found myself wanting more sophisticated filtering than demographic. They advised me to put a lot of work into my screening questions—such as asking people to answer freeform versus yes/no answers which they could “guess” to reach the incentive. </span></p>
<p><span style="font-weight: 400;">I had to run a test campaign on each platform to gauge the quality of research leads. Although they may have the smallest pool size, I found Askable to have the best quality leads with the highest engagement. (We were generally looking for full-time office workers and ran this test campaign in Australia where both Askable and we are based.)</span></p>
<h3>Asking friends and family, HR lends a hand, hiring an assistant…</h3>
<p>The next step for us was to ask around the company for referrals. I liken this to an actual startup because sometimes the first customers you try to sell to are your friends and family. While the referrals are very strong, they can run out quickly.</p>
<p>For a while, we had someone on our people and culture team who normally looks after employee recruitment put in a day a week helping us with research recruiting. I think that’s a testament to great company culture when another department can pitch in like that.</p>
<p>We ended up actually creating a part-time—3 day a week role for research assistant—to mainly focus on this recruitment function for research for all the campaigns being run across the product team.</p>
<h2>Booking</h2>
<p>We used Calendly for our booking system. What worked well here was adding our photos to the booking page, an overview of what we were trying to do and being able to explain for privacy what we’d be doing with the data (and gathering acceptance). We used a team account and Calendly checked our individual calendars for availability. I also configured my event page to send an email and SMS reminder to the candidate the morning off and 1-2 hours ahead of our meeting time. This really helped our “show up” rate.</p>
<p>What could be better in this area is screening questions upfront. Due to our recruitment, 95% of the people we’ve spoken to have been a good fit. However, we could have filtered out a few non-fits with a screener questionnaire. We’re not sure about the impact on conversions this would pose.</p>
<p>Also what was slightly problematic was not being able to conditionally route bookings to a different calendar if needed. The fine product folk at Calendly know about this and it seems their automation logic may solve this in the future.</p>
<p>Calendly has a “round-robin” feature so now certain types of interviews are booked in “round-robin” style to our whole team. It checks the individuals&#8217; calendar but interviews go into a central research calendar. This calendar is shared with the whole company. We announce certain types of interview books on Slack because we want to encourage anyone available to join the call. We’ll only take 1-2 extras, however.</p>
<h2>The interview</h2>
<p>Interview technique is a topic that’s been well covered in Mind the Product talks but we did establish some approaches and principles worth sharing.</p>
<h3>Scribing</h3>
<p>All of our bookings went into a central calendar so that one person from the product team (usually the person who launched the campaign) could be the interviewer / asking the questions and another person (anyone else from the company) could volunteer to “scribe.” We wanted to be respectful of people’s time and promised that we should be able to complete an interview in 20-30 minutes. The scribe would be taking notes live in a Google doc (live collaboration for the win) and the interviewer would be able to see the notes being taken. This approach was helped to quickly establish rapport and avoid multiple interviewers trying to dig into different areas in the interview. Sometimes the scribe would add extra questions to the document that the interviewer could ask. This helps the interviewer stay focused on the conversation.</p>
<h3>Intro</h3>
<p>To introduce the interview we’d remind the person what the company does, the purpose of our meeting being research and ask for permission to record the meeting (and begin recording if that was provided). Recording was very easy on Google meet—the recording would go directly into our Google drive where we were storing them anyway, we could easily paste the link into our company wiki. A really enjoyable workflow was the Google drive import -&gt; Clipchamp edit feature where I could quickly clip a moment from the interview and then share on Slack. Google recently adjusted meet recordings to be on a higher pricing tier so we’re back to using Quicktime + loopback (on OSX) or alternatives like Zoom.</p>
<h3>Not leading</h3>
<p>We’d have a beginning set of questions that we’d work from loosely but more so be guided by trying to understand the significance of what the interviewee was sharing.</p>
<p>Try not to ask “do you want this feature?” Ask them about past behaviour because that’s a great predictor of the future. When done right, customer interviews can help you deeply understand their beliefs, expectations, problems and context.</p>
<p>For most interviews, after completing we’d do a quick mental retro or a quick retro between the scribe and interviewer to review our interview technique for improvements next time.</p>
<h2>Analysis and follow up</h2>
<p>After the interview, we’d paste the notes into Confluence. Confluence has a “page properties report&#8221; macro so we could put a small bit of data on each interview notes page and that could be “rolled up” and summarised at a campaign level. We capture consistent data across all our interviews such as the candidates&#8217; persona, any competitors they mentioned, etc. We also capture answers to any questions that we gather for the campaign so the data can be rolled up.</p>
<p>The Dovetail solution we&#8217;ve also been trying has full transcriptions where you can tag certain quotes and build/share reports.</p>
<p>For good interviews, we’d post notable quotes, summaries and links to interview recordings in Slack for the team to consume.</p>
<p>We also wanted to be sure to follow up with the candidate. This meant sending out gift cards where applicable (managed in a spreadsheet), sending product coupon codes and requests for referrals to other interview candidates.</p>
<p>Where we’re at now is continuing to refine our research ops approach to ensure we have the systems to continue scaling and managing these processes. We’re currently considering a CRM type solution such as NetHunt, a flexible CRM integrated into Gmail. This could be helpful for us as the spreadsheets have gotten quite large.</p>
<p>You can see me talking through some of these tips during a ProductTank <a href="https://www.youtube.com/watch?v=JVKNzbJ6GdE&amp;t=2084s" target="_blank" rel="noopener noreferrer">here.</a></p>
<p>Discover more content on <a href="https://mtpdevwpe.wpengine.com/category/user-research/" target="_blank" rel="noopener noreferrer">User Research</a><a class="c-link" href="https://mtpdevwpe.wpengine.com/tag/product-analytics/" target="_blank" rel="noopener noreferrer" data-stringify-link="https://mtpdevwpe.wpengine.com/tag/product-analytics/" data-sk="tooltip_parent"> </a>or use our <a class="c-link" href="https://mtpdevwpe.wpengine.com/content-a-z/" target="_blank" rel="noopener noreferrer" data-stringify-link="https://mtpdevwpe.wpengine.com/content-a-z/" data-sk="tooltip_parent">Content A-Z</a> to find even more product management content.</p>
<div class="g g-21"><div class="g-single a-93"><a class="gofollow" data-track="OTMsMjEsMSw2MA==" href="https://mtpdevwpe.wpengine.com/product-management-training/public-workshops/communication-alignment-workshop/"><img decoding="async" src="https://mtpdevwpe.wpengine.com/wp-content/uploads/2022/04/Training-CA-layer-of-strategy-FOOTER-scaled.jpg" /></a></div></div>

Setting up research ops for customer discovery

Jase Clamp

Comments

Read next