Productish Episode 32: Explaining Product Development to the Management – With Ralf Westphal, One Man Think Tank

In our second episode of 2017, Tim and I are talking to Ralf Westphal about the difficulties he faces when trying to improve the way product development teams work – Especially in relationship to upper management.

Note: This episode of Productish is in German.

Listen to it on iTunes or SoundCloud and don’t forget to subscribe via RSS or directly on iTunes.

5 A/B Split Testing Mistakes Product Managers are responsible for

A/B Split Testing is the most powerful tool of a product manager for getting from gut feeling and HiPPO driven product decisions to actual progress based on measurable outcomes.
But while it’s comparably easy to asses a tool and implement your first testing setup, there are a couple of meta mistakes I see being made from time to time among product managers (and which I’ve made myself).

Here’s my personal selection and how to avoid them:

Sharing results too early with stakeholders

While ‘too soon’ is measured in two dimensions (more on that below), let’s just says there’s a certain point in time during which you just should not talk about the preliminary results of a running A/B test. Especially when you’re testing around a critical business KPI there are just too many dreams and visions emerging from (mostly non-technical) stakeholders – even though the test result doesn’t stand on solid ground.

In the past, I shared ongoing progress of a test we’ve implemented with my engineering team, in order to make their efforts implementing it transparent as possible. But when it came to upper management and marketing, I learned to wait a lot longer before I communicated a result. Best case is, you get approvals from two expert perspectives: A mathematical one (e.g. Digital Analyst/BI Manager) and a practical one (e.g. Director of Product or Product Management buddy).

A/B Testing for the sake of it

While A/B testing is something to aim for because of the above mentioned reasons, you shouldn’t run any testing idea which is floating through your regular stakeholder alignment meeting. While the costs of implementing and cleaning an A/B test up are important, it’s more about the environment you want to test in.

Do you actually have enough traffic flowing through your area if product in order to generate significant results within 4 weeks? Do you have the freedom to test variants which are differentiated enough from each other? Do you have the commitment from upper management to also implement the ‘unpopular’ version and they just let you A/B test in the first place because it’s a ‘modern approach’?

Solely relying on Significance Calculators

As I wrote earlier, the mathematical expert view on your test setup and results is a crucial one. But don’t let numbers alone influence your decision when a test is finished. You’ll run into scenarios in which a significance calculator will spit out a significant result after two days or so. But seriously, two days? Common sense should tell you to at least run a test through every day of the week twice in order to balance-out ‘seasonal’ impacts.
Also, significance calculators don’t know every twist and angle of your product and its dependencies like you do. So only you will be able to spot outliers and potential bugs influencing your testing setup.



 

Confusing A/B Test results with user feedback

‘Version A won, so our users like this one more!’ While this assumption doesn’t seem far-fetched, it’s quite misleading. It’s the perfect example why quantitative and qualitative testing (and research) need to go hand in hand for a complete picture.
Even though A/B testing tells you what users are doing (like clicking 5 times more likely on the yellow button instead of the red one), it doesn’t tell you why people are doing it. It’s the unfortunate discrepancy between what people say they do and what they actually do.

So, be happy about the measurable uplift your latest winning version has caused, but make sure to check back it’s mid- to long-term impact on your product by conducting some 1:1 user interviews digging into what users actually made them click more. E.g. in the worst case you just locked the users tighter into a checkout process and they had no other choice then to proceed. Again: Correct action to impact your target KPI, but it’s helpful to know the context around it.

Focussing on the wrong KPI to influence

Speaking of KPIs: When setting up your test, it’s important to only work towards an increase or decrease (e.g. when working on a churn topic) of the very metric you can directly influence with a test. Especially when implementing a test within a multi step funnel, you can never be sure to impact the very last checkout metric when only testing improvements within the address data step.
It’s intriguing to assume that all conditions before or after your area of testing stay the same, but then again you perform an A/B test to stop assuming, right?

What were you’re most impactful when designing, implementing and analysing A/B tests? Personally, my biggest learnings not only originated in practical doing but also tons of support from my former colleagues Patrick, Timo and Gudrun.

Why you always need a Plan B when you’re validating

As a posterboy Product Manager, you of course always include a ‘Validation’ phase in your outlined product discovery. Being lean-as-fuck, you aim for a fake e-mail test or a simple teaser in your existing product in order to validate customer interest of your killer feature idea. But wait, you’re not going to meet the conditions you need in order to apply your kick-ass validation idea? Puh…then I guess it’s straight do development, right?
Well, even though this story sounds a bit over the top, it provides me with the perfect trigger, why you should always have a plan b (in fact, multiple plans) in place when it comes how to validate during a product discovery.

While validation in general is broadly used, I personally like to differentiate two types of validation: Validating the user need and validating your problem-solution fit. Both kinds of validation feature opportunities to either use quantitative or qualitative methods.

So here’s some food for thought about to get creative when your default methods can’t be applied:



  • When you’re building a new product, is there a similar product sharing some characteristics with your idea live?
  • If so, are any analytics insights available?
  • If not, is quantitative data available?
  • If not, can you talk to the users of this existing products now or send them a survey?
  • When you dive right into execution, make sure to bring contrasty versions of your solutions to your usability testings.
  • If you can’t reach the users from within your target group face-to-face, reach out via e-mail and combine questions with wireframes of your solutions (make sure to randomize the order of your displayed solutions).
  • In case you don’t have access to users from within your target group at all (likely in the context of b2b or internal products), try to recruit users with common characteristics and talk to them face-to-face.

Even if you now roll your eyes and have heard some of these approaches before, I hope to at least inspire you to come up with creative alternatives to validation again and again and again…

The best Alignment Tools for Product Managers

Generating and keeping Alignment across all levels of an organization – from C-level until the single engineer – is from my perspective one of the most important (and often most challenging) for a Product Manager.
And while often suggested a simple change of habits like ‘talk more to each other’ or ‘keep a confluence page about what has been agreed’ are right, they don’t necessarily provide the right level of confidence to a Product Manager to strive at his mission.

That’s why I picked four alignment tools for Product Managers for you to grab for your next product strategy session or product discovery kick-off. Going too much into detail about how they’re used would blow the frame of this format:

Auftragsklärung

Luckily, I have been around at XING when this framework was developed. It is a mix of the influence from Stephen Bungay and Christian Becker and has been evolving through internal forces since then. Check out the full guide right here (original link is currently broken), watch my fellow Product Tank Hamburg Co-Organizer Marc speak about it attend the upcoming workshop around Auftragsklärung facilitated by Christian Becker in April at MTP Engage in Hamburg.

Intermission by Intercom

A simple but effective one pager developed from the guys at Intercom. They managed to integrate all the stuff I like the most around product development (right now): problems first, defining specific measures of success, job stories, and a clear product scope.
Would love to see some writing on the actual usage from the product teams (and stakeholders) at Intercom in the future.



The Product Tree

The best thing about this format is its strong visual aspect. It’s maybe not as sophisticated as the other frameworks above, but it focuses on feature prioritization is its biggest strength at the same time. It ruthlessly forces everybody at stake to develop a shared understanding of what to build next.
Besides, its format is perfect for keeping the aligned vision constantly visible in the team space.

‘Working Backwards’ by Amazon

While there’s no official material on this framework, it’s widely confirmed and well-known in the product manager community. It’s the result of Jeff Bezos’ approach to cut subjectivity and fluff of product development. The result is a document which shouldn’t exceed a page and a half in length which needs to be provided to meeting participants before e.g. a kick-off meeting for tackling a new product.
Never used it in ‘production’ but will do shortly.

Which format, framework or result of common sense are you using right now to create and follow-through on alignment within your organization?

Productish Episode 31: Being a Product Manager in an Inhouse Role vs. Agency Setup – With Heide Peuckert, Head at Product Njiuko

In our first episode of 2017, Tim and I are are talking to Heide Peuckert, Head of Product at Njiuko about the differences of filling a product role inhouse vs. in an agency setup.

Note: This episode of Productish is in German.

Listen to it on iTunes or SoundCloud and don’t forget to subscribe via RSS or directly on iTunes.

3 Common Misconceptions about User Testing

Meanwhile, a lot of company’s building (digital) products have embraced (or at least accepted) the positive impact user testing can have in the product development process. But just because you somehow involve users into your processes, doesn’t mean you’re doing it right.
Personally, I had the luck to get great teaching into the art of user testing from my former colleagues Britta and Anne. And while you can and have to divide user testing into its separate domains (research, deep interviewing, usability testing), I want to point out three very common misconceptions which I stumble upon in early discussions.

Focus Groups are the shit!

Nope, they’re not. Focus groups are one of the worst ways to get feedback on your product or getting to real user struggles (right behind building assumptions just on your behavior). When it comes to pointing out failures of a product, people are herd animals. They’ll follow the most common opinion in the room and will be too embarrassed to be the first ones acknowledging a mistake or flaw.
Only personal situations and a comfy and trustworthy environment will bring you closer to see the real behavior of people and hear about their real struggles.

If you want to have the room full of like-minded customers of yours as part of your product creation process, I instead recommend the format of Co-Creation Sessions.

Personas are a great way to figure out my user’s needs

While personas are already quite an improvement for user-centered products, I still think they have a huge flaw: They focus on the wrong things and distract you!

Building upon what the smart guys behind Inside Intercom have written about why Job Stories beat the format of User Stories, I want to apply to the personas framework. Conducting the cornerstones of a ‘traditional’ persona (age, hobbies, family status, hopes & dreams) takes way too much time (or is purely based on imaginative stereotypes) and doesn’t give you any solid entry points for figuring out around which problem your product should solve.
Instead, you start chasing constructed touch points in a user journey far off a users core problem (you’re able to solve). Instead, you should almost entirely focus on the context, and the central problem which will inevitably lead to a certain ‘job‘ people hire a product/workflow for. After figuring those two things out, the demographic data of your target group becomes highly irrelevant.

If you want to apply a Persona-style approach at the beginning of your product discovery anyways, I’m a strong advocate of Proto-Personas. This format will keep your efforts focussed on demographics reduced to a minimum and will include some reality checks.



We’ll just ask them what they want and how much they’re willing to pay for it

While I can’t stand the iPhone comparison when it comes to user research (‘Steve Jobs never asked anybody, whether someone wants an iPhone), there’s a certain point in it: Users are not able to articulate the right solution to their problems. This is why you are a tasked with building products which serve their needs. But to get there, you have to entirely focus on problems and not potential solutions, when talking to users before even sketching a first wire frame.

But to get there, you have to solely focus on problems and not possible solutions, when talking to users before even drawing a first wire frame. Don’t ask them what they want (even if it itches you to shout this question at them after they’ve neglected 6 of your fancy prototypes) and rather focus on what makes them happy or grumpy.
And while you’re at least able to derive some valid qualitative data from talking to users about problems, you won’t be able to when it comes to monetization. Sure, you’ll get close to figuring out a person’s most important product attributes which make them paying (e.g. quantity over quality or cross-device availability). But whether they’ll pay in the very moment for your product needs to be determined outside of that interview room (I’ll talk about quantitative validation at a later point).

Which misconceptions about user testing are you giving you a hard time to get out of people’s heads? Check out this great article at Inside Intercom when you’re looking for more mistakes.

Frequently asked product questions when building native mobile apps

Since last week’s column was the first with real ‘hands-on’ advice, I recognized that a lot of you out there are battling with quite some operational questions around building native mobile apps.
Based on the feedback I received, I wanted to answer some of your more popular questions directly in this week’s newsletter. Again, I hope that you find value in it and keep the questions coming!

iOS or Android first?
Totally depends on your target group – start with the OS which is most popular amongst your most relevant group of users (business decision). In general, I highly recommend shipping an Android and iOS version in the end, but starting with an offset of at least one or two sprints.
This way, you’ll only make mistakes regarding general logic/usability and backend connection once.


What’s iteration 0/the first spike in app development?
Very technical, but I always try to get to a first iteration of tangible user value right from the beginning. Other approaches are feasibility checks for the apps most challenging areas (technical aspects). Again: totally valid and probably a better fit for incredibly technical-focussed projects, but I personally like to shoot for the basis of an app from user perspective (e.g. login)


How to come up with a nice app icon?
Get a great Visual Designer and provide her distinct cornerstones regarding your branding and the core job you want to communicate with the app. Great design resources for creating app icons on various platforms can be found right here.


Does an app’s JTBD need to be reflected in its name?
Speaking of jobs. For answering this question, you have to distinguish between two scenarios: apps with an already strong brand and upcoming new apps (which is likely what you out there have as well). In general I always recommend to set an app name based on a users search behavior for completing a task/job. This is not only relevant for the App Store itself, but also for a quick start of your app later on when it’s accessed via e.g. Spotlight (iOS).
So, while nobody searches for ‘self destructing messages social network’ but goes for ‘Snapchat’ instead, you’re much more likely to get into a users relevant set when including ‘gas station finder’ next to your brand name (a pretty common job and App Store search term).


How to conduct remote user testing?
Personally, I’m a huge fan of invision for assembling mobile prototypes. A neat workaround for walking someone through it would be a mix of those tools (Mac/iOS focussed):

Let me know, how this setup works out for you. A decent and more straightforward solution for Android can also be found checking out lookback.

How to get better at asking for Push Notification Permissions

Inspired by a (German) video my friend Marcel published a couple of days ago, I’d like to talk about push notifications this week. While Marcel focussed on how apps sometimes carelessly misuse the precious push notification permission, I’d like to focus on one step prior to that: asking the user for the permission to send push notifications at all.

Thereby, I want to focus on 4 important aspects of getting this opt in, I often times see just being done incredibly wrong or are just huge opportunities most app creators miss out on.

Timing

You know what sucks? Getting presented with a pop up asking me for a system-wide permission before I even had the chance to get a glimpse of an app’s content. Seriously, this not only reflects bad user experience decisions but also engineering ones. Just because the app has the technical capability of sending pushes, you shouldn’t use the default timing for asking for permission right after the app has been downloaded for the first time.

Instead, bring up the permission after a moment during which the user wants to get actual value from your app and which can then be enhanced by a push notification. For example when a video on YouTube is favorited and you get asked wether you want to get reminded when new videos from that channel get published or not.

Use custom opt-ins for more chances

While you can often times use a ton of marketing mechanics in order to make a bad user experience kind of ‘unhappen’, you can’t do so with push notification permissions. Especially when you’re using the standard iOS dialogue for getting the opt-in, a ‘no’ in that very moment can’t be reversed. You won’t get a second chance to ask again – Even if you considered my tip on timing from above.

This is why I highly recommend building your own ‘pre permission’ dialogue. This doesn’t have to be fancy, but by putting (really low) effort into your own pop up, you don’t miss that one shot to get the opt in should you ask the user in the wrong moment. Instead, it’s up to your (ideally backend-steered) logic when to ask again and again. And only if you pre qualified a users intent for receiving pushes from your custom dialogue, you let her access the official iOS one. You’ll be amazed how much higher your opt in conversion rates will be.



Tracking

Speaking of conversion rates, another disadvantage of the standard iOS dialogues is the lack of analytics within that dialogue. The only rough estimations you can derive from a standard permission implementation is if the ratio of sent out push notifications (a number which should be present in your backend) increases in parallel to the number of new acquired users. A pretty muddy estimation game.

Having a custom ‘pre permission’ dialogue (I kind of start to like this term) in place also let’s you track the interaction with it in more detail – e.g. within certain user groups. You still don’t get the final numbers from the iOS system dialogue, but you’ll learn much more about when and how to ask for it.

Design

Even though I personally consider timing the much more important aspect of getting a push notification opt in, the actual design of you asking for it plays a huge role as well.

And by design I not only mean the actual Visual Design but also the copy. Customizing the push notification permission gives you the opportunity to deploy multiple variants of how to actually ask for it at the same time. AB/Multi-variant testing is an incredible opportunity by itself – But deploying it for figuring out how to unlock one of the strongest engagement mechanics of your mobile app is HUGE! Regarding the actual pixels I personally think you should just carry-through your product’s identity (like Candy Crush is clearly doing blow).

Candy Crush Push Notification Permissions

Nothing too fancy, but I think a clear differentiation from the then followed default dialogue is a plus.

So, how is your app currently asking for push notification permissions? I hope you can derive some concrete improvements for your own work. Let me know what you’ve found out!

Investing in your skills pays off in the long run

I recently had to run a two-day workshop to assess new product visions and plan the needed discovery phases accordingly. Due to some scheduling fuck-ups on my side, I hadn’t been able to put in the preparation time for the workshop I usually intend to have.
So I expected to have a rather mediocre workshop itself, not matching my typical aspiration for a fun thing like that.

Contrary to my expectations, the workshop went pretty well. We had a superb time, produced exactly the output we needed and were after all pretty aligned on the outcome.
Despite being happy with the results, I couldn’t help to think about why it worked that well, lacking my usual load of preparation time and thoughts. I almost felt guilty that I was ‘lucky’ and the scenario, day and time just played for me and led to the great results.

After putting some thoughts in it, I think it was something very simple which resulted in the situation: experience.
By doing stakeholder-heavy product work for six years and being in challenging environments managing high-scale products, I can now benefit from this year-long learning and refinement of my skills. This is what allows me to now being able to run workshops or discovery phases with fewer upfront effort.

When I arrived at that conclusion, a proper comparison came to my mind which was often articulated by my former swim coach back in the days. As you can imagine, putting a lot of mileage into swim training with a high share of endurance work’s not really what you crave for as a short distance specialist (like I was).
Observing the training plans of older world-class sprinters, which focussed much more on the intensity and had an incredibly small number of endurance sessions, we then raised the question towards our coaches why couldn’t train the same way.

The answer was simple, yet unsatisfying for me back at that time. Those athletes did put a lot of effort into laying endurance foundations when they were younger (e.g. during junior years) and could now still benefit from this high level they’ve built up for themselves. So they only had to put in quite few endurance effort right now to simply maintain this threshold.

It’s the same process which I described observing my actions – I can now benefit from the efforts and investments I put into my abilities the years before. Other examples are my efforts to ramp-up in coding, design and even online marketing to now being in a much better position to orchestrate those disciplines to form a great overall product.

The most crucial thing when arriving at this insight is though, to determine what the skills are you need to invest in today, to benefit from in a couple of years. For me, it’s, for example, public speaking and leadership – What are yours?

Why I don’t believe in ‘Idea Backlogs’

Hello and Happy New Year (just in case we haven’t seen each other yet in 2017). I hope you enjoyed the holidays and have been able to take some time off to recharge for whatever is next on your agenda.

Especially the beginning of the year we find ourselves in the middle of quarterly or even yearly planning discussions regarding which product initiatives to tackle next.
Often, the few items which find a slot on the short term roadmap are the essence of a so-called ‘long list’ of ideas. But what happens with all the other brainstormed output? They get thrown into a so-called ‘idea backlog,’ right next to that great idea you had last year and tried to bring into development. Especially when it has been a close call regarding prioritization, it feels like a good excuse for all participants to not entirely dismiss an idea but to ‘save it for later.’

Let me tell you what: Don’t do that. If an idea is just not good enough to outperform ‘competitors’ in the very moment and is so unbelievably full of value for you and your key stakeholders that you immediately want to dedicate your best resources to it right now. It won’t be any better the next time you revisit that list of broken dreams in your excel file.
Instead, I highly recommend making a clear cut below the very top product ideas which find their ways into execution and just start a new ideation process from a blank sheet the next time a roadmap slot is up for discussion.
And if an idea is truly so good that it’ll appear again and again on your lists, it’ll find its way into production mode when the time is right. Trust me.

Let me give you another example of this way of thinking: I  did the same thing when 2016 ended with all my blog post ideas and ‘books to read’ resolutions. When they were just not that good that I wanted to immediately sit down and dedicate writing or reading time to them which I otherwise probably would have spent working out or watching YouTube, I deleted them. No more drafts and plans to ‘read this book very soon.’
When a topic or book is really relevant to me, it’ll somehow resurface somewhere to me and will then get tackled.

What’s in your (personal or work-related) idea backlog? And how long have some of those items been in there? See, just kick them out and just execute on the next things which are driving you for real.

This article appeared first on my e-mail list for product people, UX designers, and entrepreneurs.