Product Score Card – How To Measure Product Performance For Emerging Products?

How do you determine that your new product initiative is successful? What performance criteria do you put in place to measure and track performance of your emerging products? Is it revenue, is it new customer acquisition or sales growth? There are no easy answers, particularly, when you are launching a new initiative within a large organization where most of the performance metrics are designed for mature products. 

In this blog, I would like to share a set of quantitative metrics and qualitative indicators which you may want to track to determine success of emerging products.

To set the context, let us take a quick look at the product maturity based on 3-Horizons of Growth (Reference: The Alchemy of Growth – Practical Insights For Building The Enduring Enterprise By Baghai, Coley and White). Your product portfolio could fall in one of the 3-growth horizons:

o   Horizon I: This represents your current business with a mature product line.  Key strategic drivers are likely to include extending the core business and defending the turf.

o   Horizon II: This is where some of the new ideas have taken shape and you are beginning to see rise in your customers and revenues with a potential upside to become a major line of business. At this stage, you still need to invest in it.

o   Horizon III: These are some of the initiatives where you have gone beyond ideation.  It’s likely that you are validating product-market fit and refining product hypotheses based on validated learning (Please feel free to refer to the blog Building Product Hypotheses And Implementing Lean Product Development Model at http://20-milesmarch.blogspot.com/2013/09/building-product-hypotheses-and.html).

For now, let us focus on a set of metrics that you want to establish to measure product performance for products which fall in Horizon II.

Most of the organizations have well established performance metrics for mature products, but there are still gaps when it comes to making investment decisions on products which fall in the emerging category.  As a result, emerging products often don’t achieve full potential because they are measured against a set of metrics which are not relevant; they remain under funded and lack management attention needed to be successful.

As a starting point, to measure performance of emerging products, here is a set of recommended quantitative metrics and qualitative indicators:

Business leaders may want to customize the list above to create top 5-metrics to measure the performance of emerging products.  For the initial releases of emerging products, it’s recommended that there is a greater emphasis on acquiring new customers vs just tracking revenues. Gaining an insight into how customers ‘consume’ the product to achieve the business outcomes is critical. It helps shape roadmap and Go-To-Market plans for the emerging products. Also, customer acquisition velocity and renewals provide an important measure of adoption and value seen by the target customers. The quantitative measure helps shape product roadmap and execution priorities by bringing greater focus around increasing adoption and value creation for the target customers and end users. Remember, your product(s) is still in the emerging phase and has not reached a level where you have a clear repeatable pattern to make a firm revenue and growth forecast.

In addition to measuring quantitative metrics, it’s also important that qualitative indicators are tracked to make product strategy and investment decisions.  Quantitative metrics alone are not sufficient to make such decisions.

Here is a recommended list of qualitative indicators that you want to capture and analyze to track performance of emerging products:

 

The following section discusses a set of probing questions to develop qualitative indicators as a part of metrics measurement process for emerging products:

·       Target customers.

o   Are we seeing traction only within the existing customers? What about non-customers? Can we get them to start using the product? Disrupters create new markets where none existed before. Great example of this is SlingTV, which is a part of Dish Networks, and is targeted at an entirely different customer segment than their traditional Satellite TV customers.

o   Is there a commonality across multiple segments? In other words, is there a set of outcomes that your product enables which cuts across multiple segments?

o   What’s your strategy to sustain core customer base while you develop new line of business around emerging products?

·       Consumption workflow.

o   How does your product get consumed by the user to achieve the expected outcome? On a side note, market creation always does not involve destruction. It may be a good idea to clarify that your product is at least as much about nondestructive creation as it is about disruption.

o   Understanding the consumption cycle is super critical for emerging products so you can define partner strategy and also identify opportunities to disrupt the market.

·       Understanding customers.

o   Do you really understand who the customer and end users are?

o   Are there opportunities to go to the actual user or you have to go through intermediaries?

o   Do you really understand the customer outcomes which matter to them? Do you clearly understand the purchase criteria?

o   Are customer satisfaction scores as high as the financial indicators?

o   Are there consumption gaps?  In other words, does your product have more features and complexity than most of your customers value?

·       Business model.

o   What are the business models which are out there today for similar products? Understanding these models is critical even if they are not attractive today.

o   Is there a greater shift happening in the industry which may change cost-value equation? How does that affect your product?

o   How durable and scalable your business model is? Are there changes in the market which could threaten your business model?

By combining quantitative metrics and qualitative indicators, you can not only establish a clear set of success criteria for the emerging products, you will also have analytics to shape your future direction to build a winning product!

Building Great Products - 4 Traits That Matter!

In my previous blog I had written about the following 5-foundational and essential themes focused on customer centricity to building a winning product.

·       Identify key customer outcomes and focus on maximizing value creation around those outcomes

·       Gain a better understanding of the consumption model.

·       Establish a clear balance between acquisition and adoption requirements.

·       Do not copy competitor’s customer acquisition strategy.

This blog is more about the core tenets essential to be a successful product leader. Product leaders with these tenets not only deliver great products but also energize their teams to achieve goals often seem unachievable in the beginning.

·       Be objective and flexible

Taking a step back, and not letting confirmed biases drive the decision making are critical. Particularly, when you are launching a new product and validating product-market fit in a crowded market, objectivity is super critical.  One win or loss is not good enough a sample size for deciding overall product direction. Reflect on what customers and product metrics are telling. Use this information to evaluate the overall strategy vs shifting gear with each sample.

 

The key is to be objective and at the same time be flexible. Not be driven by a set pattern is an important characteristic of a successful product leader. The product leaders must not surrender the game leaving any unavailable card unplayed. It’s important to keep the opportunity open to change your mind based on evidence.

 

·       Don’t get driven by journalism; rely on real-time market data that is detailed and reliable

Consistent with being objective and flexible, it’s the responsibility of the product leader to avoid product decisions which are triggered by ‘journalism’.  Often times, someone in the leadership team plays an analyst and brings up sensational items with an extreme point-of-view suggesting that either you are winning and you are the ultimate conqueror or it’s dooms day.  Watch out for such journalism shaping your product strategies and therefore execution priorities. These individuals often keep piling up information without really accomplishing anything meaningful.

Inculcate the culture and establish the processes which allow sharing of market data, in real-time, to make product decisions.  This also means that this data is analyzed in detail against a set of performance metrics to make appropriate decisions. This is critical as leaders need to be in the flow of information to be able to maintain a sense of urgency and be able to focus on what matters most.

 

·       Be a risk taker and innovative yet patient and calculating

This is the most difficult blend of science and art.  While you need to evaluate your product strategy, market fit, and the Go-To -Market plans on an ongoing basis, it’s also important that your pivot or persevere decisions are data driven and not influenced by simply copying the competition resulting into complete abandonment of your own strategy and differentiators. This could create a chaos that you cannot thrive on. In a rapidly changing market scenarios, you do have to reallocate resources to promising opportunities, but do that based on clearly defined metrics and communicate your choices to ensure agility through shared vision. In the process, make sure that there is a clear understanding of the core markets that you need to protect.

 

Being patient, assessing and recalibrating your strategy based on market evidence are essential elements of successfully managing innovation and risk in order to deliver a winning product.

·       Transparency and integrity

Building credibility by being consistent and clear when speaking to others is of utmost importance. Actions must mirror what you say. Nothing frustrates the team more than receiving mixed messages. Sense of direction is crucial at all times and more so during the times of change.

Here is a checklist in the form of probing questions you may want to start with:

·       Even when NPS (Net Promoter Score) is high, are your customers loyal to your product, or are they captive until they find a better alternative?

·       What outcomes customers care for, and how we are delivering on those outcomes? Measuring customer centric outcomes allows an objective analysis of the value created by the product.

·       Have you looked at alternate business models? Is the existing business model durable and scalable?

·       What are the alternate options available to your customers which could diminish the value delivered by your product?

·       Are you dealing with the consumption gap? In other words, do your products have more features than most of your customers value?

Customer Centric Product Life Cycle Management

How do you explain when a successful MVP (Minimum Viable Product) and a proven product-market fit result into an unsuccessful business?

The answer to this question lies in understanding and applying a customer centric approach to managing product life cycle. The following diagram represents a simple yet profound way to manage product through 4-critical phases – Acquire customers, Optimize customer experience, Retain customers and Cultivate customers.

Absence of customer centricity, mentioned above, is likely to result into a failed product despite a successful MVP and a great product-market fit.

Here is a double click on the core thought: it’s important to understand as to how do you decide whether to focus your engineering investment in support of customer acquisition strategy and/ or in optimizing customer experience?  Customer acquisition is often about adding a lot of new features to acquire new customers either in the core markets or adjacent markets. Whereas optimizing customer experience is about making sure that the customer is able to derive full value from their investment in the product. In other words, expected outcomes are delivered to customers, and they see a business value created by those outcomes. This can be measured by Net Promoter Score and referenceability of the customer base.

Optimizing customer experience in not just about the user experience as in UI, it is about gaining a greater understanding of the ‘consumption model’. How customers use the product, what are the associated systems that the product has to interoperate with to deliver the value, and what are the inhibitors (aka frictions) in the user adoption – are some of the key questions which must be answered and fully understood not only by the product managers but also by everyone touching customers, directly or indirectly, namely, sales, SE, services, support and the engineering.

Post successful MVP and product-market fit validation, every product leader finds himself/ herself at a decision fork – Should I invest engineering resources in support of customer acquisition or in optimizing customer experience? 

This is also the time when executive management of the company needs to clearly assess as to where the product truly is from a value creation standpoint.  Temptation to acquire more customers without investing in optimizing customer experience can prove pretty expensive over time.

An easy answer to the question about where to invest is to do both but then there are only a fewer resources available to do both. One extreme yet very common bias that I have found, which is also an easy trap to fall into, is to go into customer acquisition binge and not worry about optimizing customer experience until each customer deployment becomes a real pain. This is not only unfair to the early adopters who believed in the vision of the product, it results into an endless support costs and PR debacle.

These are hard choices to make for each and every company after a successful release of MVP. Lack of investment in optimizing customer experience can trigger a spate of customer escalations, which in turn can negatively impact the morale of the employees. Talented and hardworking engineering, product and customer support teams may slip into thinking that they are not winning despite burning long hours. This could ignite a negative spiral.

Product leaders and the executive management teams have to make the right choice. Here are some thoughts around how to go about making the right choice:

·         Identify key customer outcomes and focus on maximizing value creation around those outcomes

Maintain a laser focus on value created by key set of outcomes that allowed you to win your customers. Continue to raise the value for the customers around these outcomes. This will require a lot more investment of time and resources to understand how customers are consuming the product so you can optimize their experience. 

In other words, resist the temptation to add new features for customer acquisition when you have not even optimized customer experience around initial set of outcomes that you delivered.  It can create a false sense of achievement if you get many customers in the beginning as none of them will see a sustained value coming from their investment.

It is prudent to take a step back, and focus more on delivering the expected value which led to a successful acquisition of the customers to start with. Raise the bar enough to be ahead of the competition and to claim a market leadership position in what you aim to be best at in creating customer value.

·         Gain a better understanding of the consumption model.

Ensure that sales, product and engineering teams have a greater understanding of how the product is being used by the customers.  Understanding roles of various actors, outcomes that matter to them, end-to-end use case, the workflow that the product is a part of and the velocity of the usage (aka adoption) are critical to determining the state of adoption and, therefore, the need to optimize the customer experience for a greater adoption.

Also, this will allow your sales team to gain a better insight into the critical outcomes that your product must enable to improve stickiness. In the process you can cultivate customer by delivering even greater value.

·         Establish a clear balance between acquisition and adoption requirements.

Often improving the plumbing of the core product is not seen as the exciting thing but it has the most far reaching impact on the adoption of the product and customer referenceability over time.  Technical debts that one takes early on have always proven to be very expensive. 

Find the right balance. Consider applying dedicated resources to fix the plumbing issues to optimize customer experience with the product, and a set of dedicated resources to work on new features for customer acquisition. This requires that there is a roadmap for optimizing customer experience alongside enabling customer acquisition to plan for a balanced release cycle by delivering a core set per Sprint to enable both.

·         Do not copy competitor’s customer acquisition strategy.

This can be a death spiral when product leaders simply start copying competition.  While it’s crucial to learn from the competition, it’s even more important to look at your own strategy to acquire customers and delight them so they stay with you.  Here are the key probing questions to ask – Are you enabling outcomes that matter most to the customers and that value created by you is far greater than the competition? What can you do to raise the bar and delight the customers around key outcomes you are focused on? What outcomes are you enabling that competition cannot enable today? Are you adding more user friction than the competition? If so, what is your strategy to minimize that?  These are just a sample set of questions, which probably are pretty obvious.  It’s unfortunate that most product leaders and executive management teams simply gloss over the obvious and try to copy competition.

It is ideal to leverage strategy canvas (Refer: Blue Ocean Strategy – Authors: W. Chan Kim · Renée Mauborgne, and please see my blog from Sep 3-2013) to assess what outcomes matter most to the customers to develop your winning product strategy.  Products don’t simply become great by just copying!

·         Executive support.

Customer centric product life cycle management puts much greater burden on the executive management team so they can help and lead in managing customer and field organization’s expectations. They also need to ensure that they are not leading the product and engineering teams down the path which will result into incurring expensive technical debts on an ongoing basis.  There has to be a plan to pay that debt sooner rather than later, and Executive Management must support that.

A lack of executive support and poor understanding of the need to balance investment in customer acquisition and in optimizing customer experience can cost companies their fortune.  Temptation to acquire new customers often takes critical resources away from fixing the plumbing which becomes super expensive with each release and each new customer who migrates over to a new release. It also leads to engineering burn out as they spend long and non-productive hours in fixing all the problems they knew but they didn’t get the time or resources to fix. Among other things, this can also result into an exodus of top talent as no one wants to be wasting his/ her time fixing known issues!

At the end of the day, in order to have a sustainable growth and to have delighted customer experience, you need find the right balance between acquiring customers and optimizing customer experience!

                           

Building Great Products: Interpreting your product metrics

There are several metrics that a product leader can generate. The challenge is to focus on what matters the most. This can be done by applying a simple framework to evaluate various metrics for 3-core measures of success:

• Business performance.
• Value delivered to the customer.
• Customer adoption of the solution.

There is a broad set of metrics available for B2C product categories, however, I did not find them particularly helpful if you are dealing with an enterprise centric product line. Here is a recommended list of metrics for an enterprise class product. I have been using this checklist for quite some time now; you can obviously add more or drop some from the list. Hope this serves as a good baseline.

For mature products, start with the customer success metrics. And, then work backwards to get do a Root Cause Analysis (RCA) of the business and determine your go forward plan.

One advice on interpreting the metrics – Very often, interpretation of metrics becomes challenging as many stakeholders have their own confirmed biases and ‘gut feel’. Sometimes gut feels do matter but that should not be the reason why one should ignore what the metrics tell. So, the question for the product leader is how to interpret metrics to make the right choices while processing views of some of the stakeholders which are driven by confirmed biases?

Avoid the potential pitfalls in interpreting the metrics while dealing with confirmed biases of your own or other stakeholders:

• Avoid jumping to conclusions 
• Generate enough data samples to develop metrics. This will help avoid the temptation to conclude something with very little repetition. We have a propensity to constructing a favorable interpretation of the situation with minimal information.

• Guard against confirmed biases 
• Some stakeholders are likely to look for confirming evidence, defined as 'positive test strategy' by Daniel Kahneman. As a product leader one must test the hypotheses by trying to refute them instead. 
• Watch for halo effect!

• Test the hypotheses by asking difficult questions

• Make sure you gather metrics to test various hypotheses by asking complex questions (eg. Why would you use my product over what you already have? Will you buy the product if we deliver x outcome? What is my product’s Net Promoter Score?).
• Avoid substituting a difficult question with an easier one.

• Avoid overconfident judgments

• Do not depend on intuitions alone. Intuitions often deliver extreme predictions.
• Be aware of situations where you are attempting to use a part of the feedback as a kernel to decide your future course of strategy. This could lead you down a wrong path.

The art is to blend quantitative framework analysis with how human mind interprets the data to make right product choices!

Here is a simple checklist of product metrics you may want to consider for an enterprise product line; also please see 3-steps to developing action plan towards end of this blog:

Mature Product	Measure (and analysis)	Action plan
Business Performance
Elapse time from qualified lead to sale.	Use this information to analyze GTM and sales model. Assess target markets, buying center, sales model (eg. Direct vs Indirect), Product evaluation cycle, Sales frictions (eg. User Experience) etc.
Revenue actual vs plan.	To assess execution against plan. Evaluate sales coverage model, business model, and product gaps.
Revenue growth vs market growth.	Analyze your true growth!
Booking vs TAM.	This will tell you how you are doing against your BHAG goals. RCA to include sales coverage model, need to create new purchase criteria, competitive analysis including customer choice of ‘do nothing’, opportunity to create a thought leadership to evangelize and gain mind-share.
Win/ loss ratio.	To analyze why you are winning or losing. Several metrics contribute to this.  It’s still a good measure of how customers see your product against alternate choices available to them.
Segment-wise adoption.	Use this information to analyze whether you are more successful in a given market segment or for a particular use case across multiple market segments?
Value Metrics
Ratio of referenceable customers vs total number of customers.	This is crucial.  It will help make decisions on whether you still need to invest in the ‘plumbing’ of your product. It will help achieve a balance between short-term and strategic needs.
Net Promoter Score.	Same as above. It’s a great reflection of how your customers value your product. NPS results can weigh in heavily in making product prioritization decisions.
Customer rating of the delivered value around key purchase criteria enabled by the product.	This is super critical. It will help you understand value gaps, and refine your product hypotheses as you learn on the climb. Customer feedback on value gives an opportunity to focus on end-to-end use case to deliver value. It also helps identify need for new purchase criteria which your product must enable to win in the market.
Adoption Metrics
Time to deploy the product.	Figure out why it takes longer to deliver value. What can you do to minimize time-to-value-creation for your customers?
Customer rating on the usability of the product – this would include feedback on User Experience, interoperability with the eco system.	One of the most important reasons why many good products fail! Assess if usability needs more investment and focus compared to some other ‘feature creep’ that you need to address. By not resolving usability issues one is only creating a huge deficit which will be hard to recover from as product matures and increases in complexity over time.
#1 cause of customer complaints.	This is an important piece of information which is often seen as a simple reflection of product quality issues. While that may be the case, it’s essential to look beyond to determine whether it represents a bigger issue relating to poor understanding of the customer consumption model, and therefore a need to support a different use case. For example, an end user may not be able to use the product in conjunction with the eco system that he/ she has to work with because the product either restricts the feature of another application or the other app just simply does not work with your product.
# of customer complaints on installation/ deployability (install, updates, and other issues limiting customers to unlock full potential of the solution).	Same as above.
New outcomes not yet enabled by the product	This is more an art than science; it’s not a simple metrics that you can capture.  This would require a qualitative analysis of the customer use case. It enables a product leader to make a call on new feature addition, enhancement or eco system integration priorities.
Customer retention rate	Evaluate ratio of customer retention vs new customer acquisition. A lower retention rate signals significant challenges for a sustained growth.

For products which are in emerging or embryonic stage, it would make sense to focus on some of the following metrics - for a detailed analysis please see the blog:http://20-milesmarch.blogspot.com/2013/09/building-product-hypotheses-and.html

Emerging Products	Measure (and analysis)	Action plan
Business performance
Booking actual vs plan.	To assess execution against plan. Evaluate sales coverage model, business model, and product gaps.
Win/ loss ratio.	To analyze why you are winning or losing. Several metrics contribute to this.  It’s still a good measure of how customers see your product against alternate choices available to them.
Value
Ratio of referenceable customers vs total number of customers.	This will help you build the base needed for growth phase.
Net Promoter Score.	It’s a great reflection of how your customers value your product. NPS results can weigh in heavily in making persevere or pivot decision around product hypotheses you are refining.
Customer rating of the delivered value around key purchase criteria enabled by the product.	This is super critical. It will help you understand value gaps, and refine your product hypotheses as you learn on the climb. Customer feedback on value gives an opportunity to focus on end-to-end use case to deliver value. It also helps identify need for new purchase criteria which your product must enable to win in the market.
Adoption
Time to deploy the product.	Figure out why it takes longer to deliver value. What can you do to minimize time-to-value-creation for your customers?
Customer rating on the usability of the product – this would include feedback on User Experience, interoperability with the eco system.	One of the most important reasons why many good products fail! Use this information to identify areas of friction to adoption.
New outcomes not yet enabled by the product.	This is more an art than science; it’s not a simple metrics that you can capture.  This would require a qualitative analysis of the customer use case. It enables a product leader to make a call on new feature addition, enhancement or eco system integration priorities.
# New customers acquired.	Do a cohort analysis to assess features, business model and segmentation.

For embryonic stage, here is a recommended list of metrics:

Embryonic Products	Measure (and analysis)	Action plan
Business performance
# of PoC sign-ups.	Provides detailed customer insight in terms of segment, consumption pattern, outcomes customers care for etc.
Value
Customer rating of the delivered value around key purchase criteria enabled by the product.	This is super critical. It will help understand the value gaps, and refine product hypotheses as we learn on the climb. Customer feedback on value gives an opportunity to focus on end-to-end use case to deliver the stated value. It also helps identify need for new purchase criteria which the product must enable to win in the market. It provides a detailed insight into which factors drive buying decisions,
Adoption
# of free trial sign-ups.	Level of interest in the offering.
# of activations.	Value seen by the target customers.
Ratio of Activation/ paid users.	Sustained value seen by the customers.

Developing the Action Plan

Here are key considerations for developing a comprehensive action plan:

• Move from issues to choices you want to make – Evaluate various scenarios so you can test possible solutions. Avoid proving the possibility at this stage, instead specify the conditions of success – what we must have, and what’s nice to have so we can focus on what matters the most.

• Conduct rapid validations – Using the Lean Product Development Framework, conduct tests to validate customer outcomes, new features, business model and consumption models to make persevere or pivot decisions.

• Make strategic choices – Validated learning will allow you to make choices about product priorities, GTM plans, business model and routes to market.