Building Great Products: Interpreting your product metrics

There are several metrics that a product leader can generate. The challenge is to focus on what matters the most. This can be done by applying a simple framework to evaluate various metrics for 3-core measures of success:

• Business performance.
• Value delivered to the customer.
• Customer adoption of the solution.

There is a broad set of metrics available for B2C product categories, however, I did not find them particularly helpful if you are dealing with an enterprise centric product line. Here is a recommended list of metrics for an enterprise class product. I have been using this checklist for quite some time now; you can obviously add more or drop some from the list. Hope this serves as a good baseline.

For mature products, start with the customer success metrics. And, then work backwards to get do a Root Cause Analysis (RCA) of the business and determine your go forward plan.

One advice on interpreting the metrics – Very often, interpretation of metrics becomes challenging as many stakeholders have their own confirmed biases and ‘gut feel’. Sometimes gut feels do matter but that should not be the reason why one should ignore what the metrics tell. So, the question for the product leader is how to interpret metrics to make the right choices while processing views of some of the stakeholders which are driven by confirmed biases?

Avoid the potential pitfalls in interpreting the metrics while dealing with confirmed biases of your own or other stakeholders:

• Avoid jumping to conclusions 
• Generate enough data samples to develop metrics. This will help avoid the temptation to conclude something with very little repetition. We have a propensity to constructing a favorable interpretation of the situation with minimal information.

• Guard against confirmed biases 
• Some stakeholders are likely to look for confirming evidence, defined as 'positive test strategy' by Daniel Kahneman. As a product leader one must test the hypotheses by trying to refute them instead. 
• Watch for halo effect!

• Test the hypotheses by asking difficult questions

• Make sure you gather metrics to test various hypotheses by asking complex questions (eg. Why would you use my product over what you already have? Will you buy the product if we deliver x outcome? What is my product’s Net Promoter Score?).
• Avoid substituting a difficult question with an easier one.

• Avoid overconfident judgments

• Do not depend on intuitions alone. Intuitions often deliver extreme predictions.
• Be aware of situations where you are attempting to use a part of the feedback as a kernel to decide your future course of strategy. This could lead you down a wrong path.

The art is to blend quantitative framework analysis with how human mind interprets the data to make right product choices!

Here is a simple checklist of product metrics you may want to consider for an enterprise product line; also please see 3-steps to developing action plan towards end of this blog:

Mature Product	Measure (and analysis)	Action plan
Business Performance
Elapse time from qualified lead to sale.	Use this information to analyze GTM and sales model. Assess target markets, buying center, sales model (eg. Direct vs Indirect), Product evaluation cycle, Sales frictions (eg. User Experience) etc.
Revenue actual vs plan.	To assess execution against plan. Evaluate sales coverage model, business model, and product gaps.
Revenue growth vs market growth.	Analyze your true growth!
Booking vs TAM.	This will tell you how you are doing against your BHAG goals. RCA to include sales coverage model, need to create new purchase criteria, competitive analysis including customer choice of ‘do nothing’, opportunity to create a thought leadership to evangelize and gain mind-share.
Win/ loss ratio.	To analyze why you are winning or losing. Several metrics contribute to this.  It’s still a good measure of how customers see your product against alternate choices available to them.
Segment-wise adoption.	Use this information to analyze whether you are more successful in a given market segment or for a particular use case across multiple market segments?
Value Metrics
Ratio of referenceable customers vs total number of customers.	This is crucial.  It will help make decisions on whether you still need to invest in the ‘plumbing’ of your product. It will help achieve a balance between short-term and strategic needs.
Net Promoter Score.	Same as above. It’s a great reflection of how your customers value your product. NPS results can weigh in heavily in making product prioritization decisions.
Customer rating of the delivered value around key purchase criteria enabled by the product.	This is super critical. It will help you understand value gaps, and refine your product hypotheses as you learn on the climb. Customer feedback on value gives an opportunity to focus on end-to-end use case to deliver value. It also helps identify need for new purchase criteria which your product must enable to win in the market.
Adoption Metrics
Time to deploy the product.	Figure out why it takes longer to deliver value. What can you do to minimize time-to-value-creation for your customers?
Customer rating on the usability of the product – this would include feedback on User Experience, interoperability with the eco system.	One of the most important reasons why many good products fail! Assess if usability needs more investment and focus compared to some other ‘feature creep’ that you need to address. By not resolving usability issues one is only creating a huge deficit which will be hard to recover from as product matures and increases in complexity over time.
#1 cause of customer complaints.	This is an important piece of information which is often seen as a simple reflection of product quality issues. While that may be the case, it’s essential to look beyond to determine whether it represents a bigger issue relating to poor understanding of the customer consumption model, and therefore a need to support a different use case. For example, an end user may not be able to use the product in conjunction with the eco system that he/ she has to work with because the product either restricts the feature of another application or the other app just simply does not work with your product.
# of customer complaints on installation/ deployability (install, updates, and other issues limiting customers to unlock full potential of the solution).	Same as above.
New outcomes not yet enabled by the product	This is more an art than science; it’s not a simple metrics that you can capture.  This would require a qualitative analysis of the customer use case. It enables a product leader to make a call on new feature addition, enhancement or eco system integration priorities.
Customer retention rate	Evaluate ratio of customer retention vs new customer acquisition. A lower retention rate signals significant challenges for a sustained growth.

For products which are in emerging or embryonic stage, it would make sense to focus on some of the following metrics - for a detailed analysis please see the blog:http://20-milesmarch.blogspot.com/2013/09/building-product-hypotheses-and.html

Emerging Products	Measure (and analysis)	Action plan
Business performance
Booking actual vs plan.	To assess execution against plan. Evaluate sales coverage model, business model, and product gaps.
Win/ loss ratio.	To analyze why you are winning or losing. Several metrics contribute to this.  It’s still a good measure of how customers see your product against alternate choices available to them.
Value
Ratio of referenceable customers vs total number of customers.	This will help you build the base needed for growth phase.
Net Promoter Score.	It’s a great reflection of how your customers value your product. NPS results can weigh in heavily in making persevere or pivot decision around product hypotheses you are refining.
Customer rating of the delivered value around key purchase criteria enabled by the product.	This is super critical. It will help you understand value gaps, and refine your product hypotheses as you learn on the climb. Customer feedback on value gives an opportunity to focus on end-to-end use case to deliver value. It also helps identify need for new purchase criteria which your product must enable to win in the market.
Adoption
Time to deploy the product.	Figure out why it takes longer to deliver value. What can you do to minimize time-to-value-creation for your customers?
Customer rating on the usability of the product – this would include feedback on User Experience, interoperability with the eco system.	One of the most important reasons why many good products fail! Use this information to identify areas of friction to adoption.
New outcomes not yet enabled by the product.	This is more an art than science; it’s not a simple metrics that you can capture.  This would require a qualitative analysis of the customer use case. It enables a product leader to make a call on new feature addition, enhancement or eco system integration priorities.
# New customers acquired.	Do a cohort analysis to assess features, business model and segmentation.

For embryonic stage, here is a recommended list of metrics:

Embryonic Products	Measure (and analysis)	Action plan
Business performance
# of PoC sign-ups.	Provides detailed customer insight in terms of segment, consumption pattern, outcomes customers care for etc.
Value
Customer rating of the delivered value around key purchase criteria enabled by the product.	This is super critical. It will help understand the value gaps, and refine product hypotheses as we learn on the climb. Customer feedback on value gives an opportunity to focus on end-to-end use case to deliver the stated value. It also helps identify need for new purchase criteria which the product must enable to win in the market. It provides a detailed insight into which factors drive buying decisions,
Adoption
# of free trial sign-ups.	Level of interest in the offering.
# of activations.	Value seen by the target customers.
Ratio of Activation/ paid users.	Sustained value seen by the customers.

Developing the Action Plan

Here are key considerations for developing a comprehensive action plan:

• Move from issues to choices you want to make – Evaluate various scenarios so you can test possible solutions. Avoid proving the possibility at this stage, instead specify the conditions of success – what we must have, and what’s nice to have so we can focus on what matters the most.

• Conduct rapid validations – Using the Lean Product Development Framework, conduct tests to validate customer outcomes, new features, business model and consumption models to make persevere or pivot decisions.

• Make strategic choices – Validated learning will allow you to make choices about product priorities, GTM plans, business model and routes to market.