The Art Of Product Management - Blending Quantitative Framework With How We Think

In my previous blogs I emphasized the importance of using a metrics driven framework to build a set of hypotheses, then test and develop the product using lean product development approach to deliver the outcomes that the customers value. 

Very often, interpretation of metrics becomes challenging as many stakeholders have their own confirmed biases and ‘gut feel’. Sometimes gut feels do matter but that should not be the reason why one should ignore what the metrics tell. So, the question for the product leader is how to interpret metrics to make the right choices while processing views of some of the stakeholders which are driven by confirmed biases?

While reading an excellent book (gifted to me by a good friend and colleague of mine), Thinking Fast and Slow by Daniel Kahneman, I learned several lessons that can be applied to making right product decisions.

Inspired by the book, here is my checklist to avoid pitfalls in interpreting the metrics while dealing with confirmed biases of your own or other stakeholders:

1.       Avoid jumping to conclusions

·         Generate enough data samples to develop metrics. This will help avoid the temptation to conclude something with very little repetition. We have a propensity to constructing a favorable interpretation of the situation with minimal information.

2.       Guard against confirmed biases

·         Some stakeholders are likely to look for confirming evidence, defined as 'positive test strategy' by Daniel Kahneman. As a product leader one must test the hypotheses by trying to refute them instead.

·         Watch for halo effect!

3.       Test the hypotheses by asking difficult questions

·         Make sure you gather metrics to test various hypotheses by asking complex questions (eg. Why would you use my product over what you already have? Will you buy the product if we deliver x outcome? What is my product’s Net Promoter Score?).

·         Avoid substituting a difficult question with an easier one.

4.       Avoid overconfident judgments

·         Do not depend on intuitions alone. Intuitions often deliver extreme predictions.

·         Be aware of situations where you are attempting to use a part of the feedback as a kernel to decide your future course of strategy. This could lead you down a wrong path.

The art is to blend quantitative framework analysis with how human mind interprets the data to make right product choices!