Why Retesting your Experiments is crucial for real Optimization Impact

Did you run AB tests before during your career as a product person?
Good.

Did you manage to achieve significant results with at least one of those tests?
Great.

Did you reject the same idea when a colleague suggests it as an experiment 6+ months after you ran it?
Not good. Wait, what?

What might seem counterintuitive at first, is one of the most common mistakes conversion optimizers make throughout their careers.
The primary motivation for testing hypotheses using AB tests is to gather data to avoid decisions which are solely based on gut feeling.
You also want to reduce waste by preventing colleagues in your company from putting effort into a testing idea you already invalidated.

But what if you’re causing more harm than good with this behavior? I recently had the chance to attend a talk from Willem Isbrucker, Senior Product Owner at Booking.com during our Product Tank Hamburg event and got a valuable lesson taught.

When he presented the audience with an AB test setup he ran and asked the audience for the winner, 80% in the room predicted the right version (variant). But then he revealed that the 20% who considered the control version to win was also right in a certain way. The experiment also ran a year ago with the opposite result.
While you could assume a mistake in the analytics, the real reason behind it was that the environment around the testes element (a search box) evolved, and thereby the element itself performed better being visualized differently.

And while looking back at my own set of AB tests I ran throughout my career, I also rejected ideas of retesting because…well, we already proved the hypothesis to be wrong or right.

So, whenever you get presented with a testing idea which ran 6+ months in the past, look again at your product and check the situation:
Did you make changes to the main navigation in the meantime? Did you test within a particular segment or across the user base? Does it maybe make sense to target the change at a particular user group? What if the recently introduced new pricing tier would impact the test again?

Try to find a balance between reducing waste and embracing retesting of known hypotheses. You might be surprised regarding the impact you can generate.

How amazon prime does Churn Prevention

I looked at the cancellation process of the amazon prime membership to find out how amazon tries to prevent its prime members from churning.

You can find the complete transcript of my teardown right below.

  1. Intro Slide
  2. So, I want to cancel my amazon prime membership. Where to start?
  3. Maybe over here?
  4. It’s an ‘Account-ish’ thing, I guess?
  5. Ah, right in the first row of options. Very handy.
  6. So much prime stuff. Bet they’ve hidden the cancelation option as deeply as possible…
  7. Oh, look. Quite accessible and neatly organizes with other account related settings. Nice.
  8. I love progress indicators. They’re great for setting user expectations at the beginning of a process (honestly, I’d have expected a longe one here).
  9. Ok, obviously bad timing for this teardown. Prime Day is around the corner and obviously a huge counter argument.
  10. But what do I see here? Three equally weighted buttons? Very un-amazonian. Let’s go through them.
  11. A reminder? That sounds fair and handy. Let’s check out this option
  12. Back where I started with a subtle but comprehensive confirmation. Great option as I can’t drop the membership much earlier anyways.
  13. In general, meanwhile ‘standard’ practice for cancelation copy applied right here: Speaking in (losing) benefits. This one just lets me drop out of the funnel.
  14. Ok, let’s say I don’t want to keep those amazing benefits…
  15. Interesting that they’re aiming this counter argument solely towards pricing. They didn’t ask for any reasons before?
  16. Interesting that they’re aiming this counter argument solely towards pricing. They didn’t ask for any reasons before? Probably they learned over time that this is the only main reason for people to cancel prime. Therefore, it makes perfect sense.
  17. Interesting that they’re aiming this counter argument solely towards pricing. They didn’t ask for any reasons before? Probably they learned over time that this is the only main reason for people to cancel prime. Therefore, it makes perfect sense. I also really like that they don’t work with discounts to target ‘my’ pricing-related cancelation reason. It would depreciate the value of the membership
  18. I still get the known options to receive a reminder or drop out of the funnel.
  19. So, is this already the final button to end the membership? Reads like it, but the progress indicator shows one more step…
  20. Ah, of course not. This is the final final call. Neat overview of the relevant facts (end date) and the known options.
  21. For some reason, the buttons are still equally weighted. Not because of ‘dark patterns’, but just more better user orientation.
  22. Clear and obvious confirmation of my cancelation and when I’ll lose access to my prime benefits.
  23. I like the (maybe too) subtle option to directly reverse my decision. Not screaming for attention but perfectly integrated.

My Recap of the Working Products 2017 Conference

While Germany overall might be a bit short on high-quality product management conferences, Hamburg is certainly not.
Not that long after the great MTP Engage conference took place, the Working Products started into their second edition, gathering passionate product and design people for a two great day exchange of thoughts and knowledge.
The main difference for me personally was that I also delivered a talk about how and when MVPs are too expensive in the context of validation (more on that in a later post).

Format & Organization

The event was held in the facilities of eparo, a UX consulting agency which is also the organizer of the format. Rolf and his team did a great job leveraging the available space for the conference format.

What’s interesting about the Working Products was this year’s organizational schedule: While the ‘formal’ talks were limited to the mornings, the afternoons dedicated to the interactive and collaborative ProductSpace sessions.
This distinction was a welcomed change from traditional conference formats, and it was Alsop attractive enough to keep most of the speakers around for time beyond their keynotes.

As all talks were held in parallel, one always had to choose between Track 1 or 2. Thankfully, the choice which talks to attend wasn’t a limited one, as all presentations were also recorded.
Here are some shared impressions and thoughts from the ones I attended.

Matthias Schrader

The opening keynote was held my Matthias Schrader, CEO of the agency SinnerSchrader. While he was a bit late due to a calendar mix-up, he delivered an interesting talk based on his recent book ‘Transformationale Produkte‘.
As I haven’t read the book yet, it was nice to get a glimpse into its content and visualization. And while I’m probably not the target group for the book, I would highly recommend it to companies facing challenges due to digital transformation as it lays out the underlying mechanics instead of stopping after the glossy surface. It might spare you the next trip to Silicon Valley.

Working Products 2017 - Matthias Schrader



Heide Peuckert

Heide is Head of Product at Nijuko, a development agency based in Hamburg. I previously interviewed her at my German podcast Productish about what it takes to be a PO in a contractor environment.
She gave some extended insights into her daily challenges and struggles and pointed some key pain points out when it comes to working with clients – Some examples were the lack of overall product vision or missing commitment to invest time and money in design, leave alone user research.

Working Products 2017 - Heide Peuckert

Tim Rudolph

Tim was the first real ‘corporate’ guy speaking at the event. He’s leading the digital lab of the logistics company Hermes. And while he couldn’t share that many details regarding the actual output the unit was providing, it was fun to see how they structure their division and which challenges they faced when trying to bring ideas out to the ‘real’ world.
Particularly in the real world business of delivering packages, in which you can’t easily roll an AB test back in case of a missing legal requirement.

Roman Pichler

Roman Pichler is probably known to the majority of German product people out there. Especially for me, he’s kind of a role model who profoundly influenced my appreciation for agile product ownership through his writing and shared thoughts.
So it was evident that I had to attend the talk he gave to open day 2 of the conference. His talk was heavily based on his recent book ‘Strategize‘ which is part of my constantly growing library of product books.

If you’ve read the book, the talk didn’t add much new stuff to it, but it was impressive to experience Roman live and in action.

Working Products 2017 - Roman Pichler

Florian Grote

The last talk I attended was from Florian who shared incredible transformational insights from the music hard- and software company Native Instruments.
While I’m not a huge musician myself, I could imagine the challenges setting up discovery and delivery tracks for such complex, yet intertwined products.

Working Products 2017 - Florian Grote

Summary of the Working Products 2017

My positive impressions of the Working Products 2016 was only confirmed by my attendance of 2017. I love the theoretical and practical balance of the format and have to give a huge compliment to the catering and overall organization.
Despite feeling like a small XING alumni meetup, complemented by some ‘other people,’ I had great chats, and it was an incredible pleasure to finally getting back into public speaking. If you have an event to recommend for giving a product-related talk, feel free to suggest one or refer me.
(Almost) all slides are already available on the homepage, and the video recordings should follow in a couple of days. I curious about the lineup and location of the Working Products 2018 – See you there!

Alexa, what’s the definition of a Product Discovery?

As most of you know, the best way to understand a new technology is to work with it, instead of only using it. This was true for me during my personal ‘peak mobile’ phase, when I build an iOS app myself and has been the trigger for all of my side projects.

So while the Amazon Echo failed to cover a personal use case in my home, I remained curious whether I may have missed something which would let me acknowledge the real potential of the platform.

This is why I immediately jumped on the idea of building my very own Alexa Skill once I saw the excellent team at treehouse offering a course for it.
It didn’t take long for me to adapt the underlying idea of the course to a Skill which would work for my domain of expertise.
So I went ahead and built the Product Management Dictionary Skillfor the Amazon Echo and other Alexa-enabled devices. Go ahead and give it a try

So far, I kept the scope of this initial release very straightforward and only included definitions for three product management terms and focussed on English as a system language (this is why it may not be possible to enable the Skill on your German Echo).

I’d love to hear your thoughts on this kind of adoption for Amazon Echo capabilities. Even though the primary job of this Skill was to let me dive deeper into voice products and the Alexa developer environment, I’d love to iterate this product into something serving broader needs within the product community.

If you want to suggest another term be included in the Skill’s dictionary, just go ahead and add it to this Spreadsheet.

What to learn from Audible to avoid Payer Churn

After focussing some broader advice for product people in the past couple of weeks, I wanted to be very practical with this post by sharing a best practice regarding churn prevention: Audible.

I recently noticed that my Audible subscription got renewed besides me currently not being able to catch up on further audio books.

By completing the necessary steps to cancel it, I couldn’t help but admire how well this was done. But not only from the viewpoint of an optimizer but also from a user’s perspective.
It just felt…worshipped.

So, without further ado, here are four steps of churn prevention worth memorizing. Unfortunately, my language has been set to German, and I didn’t think of switching it in time. I’ll give some context in English in the captions to help most of you out.

Audible Churn Prevention - Step 1
Ok, nice. You’re repeating what I’m missing out by canceling now (which is a credit of three free audiobooks per month) and provide some recommendations (which are pretty much on point) for what to spend it on.
Bonus Point: Primary CTA is a positive one (‘Redeem Credit’), and you have to look twice to continue with the cancellation.
Audible Churn Prevention - Step 2
Oldie but Goodie: Asking why I want to cancel. The available options seem to be quite a lot but need to capture a lot of possibilities as well. I’m curious if the next page will react to my choice…
They’re now also dropping the CTA confusion and offer only one button to proceed with the process. Very honest and straightforward.


Audible Churn Prevention - Step 4
Subtle, but positive confirmation of my cancellation and I’m back on the page where it all started – My account settings. Well done, Audible!

Would you like to see more from that kind of teardowns from me? Specifically not on onboardings but rather not so popular processes like churn.
If so, just send me an e-mail and let me know which product or service you’d like to see assessed.

Why Product Managers need to embrace Lateral Leadership

Audio Version:

I recently got some food for thought for relevant challenges in product management from a fellow product owner who reported on the following situation:

‘The development team didn’t accept the problem I gave them to find a solution for.’
While I wouldn’t say that that’s a very common challenge for product people, the skill it requires to solve is among the top 3 qualifications it takes to succeed as a product manager: Lateral Leadership.

What is Lateral Leadership?

In short, lateral leadership means leading other people without having hierarchical power. As mentioned, it’s the exact opposite of hierarchical or top-down leadership which is (unfortunately) the most common way of managing.
And even though lateral leadership was by no means originally developed to fit the leadership position product managers are in, it’s exactly what you need to embrace if you want to lead team members and stakeholders which don’t report to you.

Why is Lateral Leadership relevant and how to apply it?

To understand why product managers are lateral leaders instead of hierarchical leaders, I want to emphasize how the company structure around product managers is usually structured:

Product Managers lateral leadership organizations

What you see here, is the result of a so-called ‘matrix organization.’ Even though (SCRUM) teams are intended to work as a unit by itself, the team members taking care of engineering and design are most often only ‘led’ to the product owner within the construct of a development team.
They all have other people managers who are overseeing the respective departments across multiple development teams and sometimes even business units.



What that means is that your team needs to accept you as the one who defined what is done, but someone else is responsible for the how.
And that’s where the point of lateral leadership comes in. While the heads of department have managerial tools to lead people (even though that should be the exception), you have to convince your team members to follow you without administrative ‘super powers’

Quite the contrary, key factors for achieving lateral leadership are furthermore things like:

  • Strong and explicit communication
  • A clear vision
  • Empathy
  • Passion
  • Trust

There’s an excellent book by Daniel Pink called ‘To sell is Human.’ In it, Pink even refers to this skill as the ability to “move” people from one mindset to another.
Successful product people need to spend time and effort on these ‘moving’ activities, bringing everyone together around a shared understanding of the customer or business problem so that everyone can be involved in helping solve it to further the business goals.

Sounds familiar? Good, then you recognized the pattern of alignment. Closing the loop on the fundamental challenge the product owner faced, a likely reason for a development team not accepting a task is the lack of context. More precisely the context around why this task matters in the greater scheme of things and how it’s execution moves the needle.

Even though you may have paid attention to alignment using one of the several available tools, I highly recommend either revisiting the initial alignment document with the team or even going back to the drawing board to get everyone’s motivational drivers considered.

What to do when Lateral Leadership tools fail?

Of course, alignment and other lateral leadership techniques can only do so much. If everything fails, I recommend 1:1 discussions with the individual team members to get to the bottom of what’s causing the resistance.
Maybe you get a handle on individual factors (e.g. an unresolved conflict from a personal situation), or you simply have to involve the respective heads of department for advice or even escalation.

Why Stakeholders mostly talk in Feature Requests

Almost every product manager knows the situation (if you don’t, please let me understand how you avoid it right from the beginning): Some of your stakeholders just always come up with very specific features (aka solutions and not problems) they want you to build into your product.
While some of them may have learned and agreed to the concept of building an MVP first and no stuffing the first product release with more functionality, they a least insist on storing them as they are in the product/feature/idea backlog (here’s why that’s not an approach to pursue).

That’s because they’re like users – They don’t know what they want. And talking about particular features is their way to remain involved in the product development process continuously.

The key for you as a Product Manager is to determine where the urge of some stakeholders to be a regular part of the day-to-day product development process originates. You have to reverse engineer your stakeholder’s feature requests back to ‘jobs‘ they want you to fulfill for them or your target user.
The most likely reason though is that they think hat the product output becomes proportionally better with their involvement.



Unfortunately, this misconception is the result of a lack of trust these stakeholders have that the delivered output by you and your development team matches their expectations.
Why that may seem like a very unfortunate situation, the best way to resolve this issue is to focus on alignment early on in the product discovery and the commitment to a particular outcome rather than feature idea.

A better way for stakeholders could be e.g. to drop in inspiring snapshots they’ve seen in other products with the apparent accompaniment that the way this product approached an individual problem may be helpful for your work.

Remain the most clueless person in the room – But be the most enlightened one

Being a Product Manager is not about being the ingenious front singer and songwriter of a band everybody adores. It’s more about being the quiet but confident maestro orchestrating an orchestra of proven experts (without the usual top-down attitude).
As a result, a Product Manager is also not the right person to name the next shiny killer feature. You’re rather the one who shows the path and sets the stage for discovering it together.

Over time, I became deeply convinced that great Product Managers are no single domain experts. Furthermore, an in general considered ‘good’ Product Manager can succeed across a vast playing field of areas – whether it’s e-commerce, finance, mobile apps or machine learning projects.
There’s even a certain risk in being too deeply intertwined with the domain you’re tackling. By having deep domain knowledge, it’s so much easier to get caught up in seemingly great feature ideas. By having a certain distance or even naivety towards the industry, you’re much better suited to avoid confirmation bias.



And by relying on your education about user-centered product development, you’re more than set to build a strikingly successful product in whichever domain you want.
Without the need of building up too much domain knowledge which, in an era of ongoing disruption, seems nowadays like unnecessary waste anyways.

Guidelines for measuring success in Product Management

When I started out in Product Management, the role, in general, was considered to be a bit more stakeholder-focussed project manager. It was all about facilitating requirements and keeping the (mostly outsourced) ‘IT department’ busy for a low daily rate.

The main success criteria to measure Product Managers again back then was basically to ship a particular feature (mostly determined by a HiPPO) in a given time and extremely budget. How essential or even reasonable a feature was, leave alone measuring the impact it may generate, were seen as unimportant factors.

Time Quality Budget Product Manager

Thanks to the influence and popularity of product-centric companies like Facebook, LinkedIn, Airbnb or XING, this perspective is now dismissed from modern product development.
Instead, Product Managers are empowered to truly take responsibility for that they were building in strong collaboration with design and development teams.



Comparing the guidance provided to Product Managers for moving forward with the “time-quality-budget“ approach, it’s now more something like:
Build the right feature for proven user needs with the leanest approach to generate a significant impact which is in line with pre-defined KPIs.

Modern Product Management Success

This lens of looking at the success of a product manager is not only important for individual people managers but also the product manager herself.
Don’t get caught up in “time-quality-budget” madness when it comes to reflecting your performance. Take a smarter approach looking at your work.