Product Score Card – How To Measure Product Performance For Emerging Products?

How do you determine that your new product initiative is successful? What performance criteria do you put in place to measure and track performance of your emerging products? Is it revenue, is it new customer acquisition or sales growth? There are no easy answers, particularly, when you are launching a new initiative within a large organization where most of the performance metrics are designed for mature products. 

In this blog, I would like to share a set of quantitative metrics and qualitative indicators which you may want to track to determine success of emerging products.

To set the context, let us take a quick look at the product maturity based on 3-Horizons of Growth (Reference: The Alchemy of Growth – Practical Insights For Building The Enduring Enterprise By Baghai, Coley and White). Your product portfolio could fall in one of the 3-growth horizons:

o   Horizon I: This represents your current business with a mature product line.  Key strategic drivers are likely to include extending the core business and defending the turf.

o   Horizon II: This is where some of the new ideas have taken shape and you are beginning to see rise in your customers and revenues with a potential upside to become a major line of business. At this stage, you still need to invest in it.

o   Horizon III: These are some of the initiatives where you have gone beyond ideation.  It’s likely that you are validating product-market fit and refining product hypotheses based on validated learning (Please feel free to refer to the blog Building Product Hypotheses And Implementing Lean Product Development Model at http://20-milesmarch.blogspot.com/2013/09/building-product-hypotheses-and.html).

For now, let us focus on a set of metrics that you want to establish to measure product performance for products which fall in Horizon II.

Most of the organizations have well established performance metrics for mature products, but there are still gaps when it comes to making investment decisions on products which fall in the emerging category.  As a result, emerging products often don’t achieve full potential because they are measured against a set of metrics which are not relevant; they remain under funded and lack management attention needed to be successful.

As a starting point, to measure performance of emerging products, here is a set of recommended quantitative metrics and qualitative indicators:

Business leaders may want to customize the list above to create top 5-metrics to measure the performance of emerging products.  For the initial releases of emerging products, it’s recommended that there is a greater emphasis on acquiring new customers vs just tracking revenues. Gaining an insight into how customers ‘consume’ the product to achieve the business outcomes is critical. It helps shape roadmap and Go-To-Market plans for the emerging products. Also, customer acquisition velocity and renewals provide an important measure of adoption and value seen by the target customers. The quantitative measure helps shape product roadmap and execution priorities by bringing greater focus around increasing adoption and value creation for the target customers and end users. Remember, your product(s) is still in the emerging phase and has not reached a level where you have a clear repeatable pattern to make a firm revenue and growth forecast.

In addition to measuring quantitative metrics, it’s also important that qualitative indicators are tracked to make product strategy and investment decisions.  Quantitative metrics alone are not sufficient to make such decisions.

Here is a recommended list of qualitative indicators that you want to capture and analyze to track performance of emerging products:

 

The following section discusses a set of probing questions to develop qualitative indicators as a part of metrics measurement process for emerging products:

·       Target customers.

o   Are we seeing traction only within the existing customers? What about non-customers? Can we get them to start using the product? Disrupters create new markets where none existed before. Great example of this is SlingTV, which is a part of Dish Networks, and is targeted at an entirely different customer segment than their traditional Satellite TV customers.

o   Is there a commonality across multiple segments? In other words, is there a set of outcomes that your product enables which cuts across multiple segments?

o   What’s your strategy to sustain core customer base while you develop new line of business around emerging products?

·       Consumption workflow.

o   How does your product get consumed by the user to achieve the expected outcome? On a side note, market creation always does not involve destruction. It may be a good idea to clarify that your product is at least as much about nondestructive creation as it is about disruption.

o   Understanding the consumption cycle is super critical for emerging products so you can define partner strategy and also identify opportunities to disrupt the market.

·       Understanding customers.

o   Do you really understand who the customer and end users are?

o   Are there opportunities to go to the actual user or you have to go through intermediaries?

o   Do you really understand the customer outcomes which matter to them? Do you clearly understand the purchase criteria?

o   Are customer satisfaction scores as high as the financial indicators?

o   Are there consumption gaps?  In other words, does your product have more features and complexity than most of your customers value?

·       Business model.

o   What are the business models which are out there today for similar products? Understanding these models is critical even if they are not attractive today.

o   Is there a greater shift happening in the industry which may change cost-value equation? How does that affect your product?

o   How durable and scalable your business model is? Are there changes in the market which could threaten your business model?

By combining quantitative metrics and qualitative indicators, you can not only establish a clear set of success criteria for the emerging products, you will also have analytics to shape your future direction to build a winning product!

Building Great Products - 4 Traits That Matter!

In my previous blog I had written about the following 5-foundational and essential themes focused on customer centricity to building a winning product.

·       Identify key customer outcomes and focus on maximizing value creation around those outcomes

·       Gain a better understanding of the consumption model.

·       Establish a clear balance between acquisition and adoption requirements.

·       Do not copy competitor’s customer acquisition strategy.

This blog is more about the core tenets essential to be a successful product leader. Product leaders with these tenets not only deliver great products but also energize their teams to achieve goals often seem unachievable in the beginning.

·       Be objective and flexible

Taking a step back, and not letting confirmed biases drive the decision making are critical. Particularly, when you are launching a new product and validating product-market fit in a crowded market, objectivity is super critical.  One win or loss is not good enough a sample size for deciding overall product direction. Reflect on what customers and product metrics are telling. Use this information to evaluate the overall strategy vs shifting gear with each sample.

 

The key is to be objective and at the same time be flexible. Not be driven by a set pattern is an important characteristic of a successful product leader. The product leaders must not surrender the game leaving any unavailable card unplayed. It’s important to keep the opportunity open to change your mind based on evidence.

 

·       Don’t get driven by journalism; rely on real-time market data that is detailed and reliable

Consistent with being objective and flexible, it’s the responsibility of the product leader to avoid product decisions which are triggered by ‘journalism’.  Often times, someone in the leadership team plays an analyst and brings up sensational items with an extreme point-of-view suggesting that either you are winning and you are the ultimate conqueror or it’s dooms day.  Watch out for such journalism shaping your product strategies and therefore execution priorities. These individuals often keep piling up information without really accomplishing anything meaningful.

Inculcate the culture and establish the processes which allow sharing of market data, in real-time, to make product decisions.  This also means that this data is analyzed in detail against a set of performance metrics to make appropriate decisions. This is critical as leaders need to be in the flow of information to be able to maintain a sense of urgency and be able to focus on what matters most.

 

·       Be a risk taker and innovative yet patient and calculating

This is the most difficult blend of science and art.  While you need to evaluate your product strategy, market fit, and the Go-To -Market plans on an ongoing basis, it’s also important that your pivot or persevere decisions are data driven and not influenced by simply copying the competition resulting into complete abandonment of your own strategy and differentiators. This could create a chaos that you cannot thrive on. In a rapidly changing market scenarios, you do have to reallocate resources to promising opportunities, but do that based on clearly defined metrics and communicate your choices to ensure agility through shared vision. In the process, make sure that there is a clear understanding of the core markets that you need to protect.

 

Being patient, assessing and recalibrating your strategy based on market evidence are essential elements of successfully managing innovation and risk in order to deliver a winning product.

·       Transparency and integrity

Building credibility by being consistent and clear when speaking to others is of utmost importance. Actions must mirror what you say. Nothing frustrates the team more than receiving mixed messages. Sense of direction is crucial at all times and more so during the times of change.

Here is a checklist in the form of probing questions you may want to start with:

·       Even when NPS (Net Promoter Score) is high, are your customers loyal to your product, or are they captive until they find a better alternative?

·       What outcomes customers care for, and how we are delivering on those outcomes? Measuring customer centric outcomes allows an objective analysis of the value created by the product.

·       Have you looked at alternate business models? Is the existing business model durable and scalable?

·       What are the alternate options available to your customers which could diminish the value delivered by your product?

·       Are you dealing with the consumption gap? In other words, do your products have more features than most of your customers value?

Customer Centric Product Life Cycle Management

How do you explain when a successful MVP (Minimum Viable Product) and a proven product-market fit result into an unsuccessful business?

The answer to this question lies in understanding and applying a customer centric approach to managing product life cycle. The following diagram represents a simple yet profound way to manage product through 4-critical phases – Acquire customers, Optimize customer experience, Retain customers and Cultivate customers.

Absence of customer centricity, mentioned above, is likely to result into a failed product despite a successful MVP and a great product-market fit.

Here is a double click on the core thought: it’s important to understand as to how do you decide whether to focus your engineering investment in support of customer acquisition strategy and/ or in optimizing customer experience?  Customer acquisition is often about adding a lot of new features to acquire new customers either in the core markets or adjacent markets. Whereas optimizing customer experience is about making sure that the customer is able to derive full value from their investment in the product. In other words, expected outcomes are delivered to customers, and they see a business value created by those outcomes. This can be measured by Net Promoter Score and referenceability of the customer base.

Optimizing customer experience in not just about the user experience as in UI, it is about gaining a greater understanding of the ‘consumption model’. How customers use the product, what are the associated systems that the product has to interoperate with to deliver the value, and what are the inhibitors (aka frictions) in the user adoption – are some of the key questions which must be answered and fully understood not only by the product managers but also by everyone touching customers, directly or indirectly, namely, sales, SE, services, support and the engineering.

Post successful MVP and product-market fit validation, every product leader finds himself/ herself at a decision fork – Should I invest engineering resources in support of customer acquisition or in optimizing customer experience? 

This is also the time when executive management of the company needs to clearly assess as to where the product truly is from a value creation standpoint.  Temptation to acquire more customers without investing in optimizing customer experience can prove pretty expensive over time.

An easy answer to the question about where to invest is to do both but then there are only a fewer resources available to do both. One extreme yet very common bias that I have found, which is also an easy trap to fall into, is to go into customer acquisition binge and not worry about optimizing customer experience until each customer deployment becomes a real pain. This is not only unfair to the early adopters who believed in the vision of the product, it results into an endless support costs and PR debacle.

These are hard choices to make for each and every company after a successful release of MVP. Lack of investment in optimizing customer experience can trigger a spate of customer escalations, which in turn can negatively impact the morale of the employees. Talented and hardworking engineering, product and customer support teams may slip into thinking that they are not winning despite burning long hours. This could ignite a negative spiral.

Product leaders and the executive management teams have to make the right choice. Here are some thoughts around how to go about making the right choice:

·         Identify key customer outcomes and focus on maximizing value creation around those outcomes

Maintain a laser focus on value created by key set of outcomes that allowed you to win your customers. Continue to raise the value for the customers around these outcomes. This will require a lot more investment of time and resources to understand how customers are consuming the product so you can optimize their experience. 

In other words, resist the temptation to add new features for customer acquisition when you have not even optimized customer experience around initial set of outcomes that you delivered.  It can create a false sense of achievement if you get many customers in the beginning as none of them will see a sustained value coming from their investment.

It is prudent to take a step back, and focus more on delivering the expected value which led to a successful acquisition of the customers to start with. Raise the bar enough to be ahead of the competition and to claim a market leadership position in what you aim to be best at in creating customer value.

·         Gain a better understanding of the consumption model.

Ensure that sales, product and engineering teams have a greater understanding of how the product is being used by the customers.  Understanding roles of various actors, outcomes that matter to them, end-to-end use case, the workflow that the product is a part of and the velocity of the usage (aka adoption) are critical to determining the state of adoption and, therefore, the need to optimize the customer experience for a greater adoption.

Also, this will allow your sales team to gain a better insight into the critical outcomes that your product must enable to improve stickiness. In the process you can cultivate customer by delivering even greater value.

·         Establish a clear balance between acquisition and adoption requirements.

Often improving the plumbing of the core product is not seen as the exciting thing but it has the most far reaching impact on the adoption of the product and customer referenceability over time.  Technical debts that one takes early on have always proven to be very expensive. 

Find the right balance. Consider applying dedicated resources to fix the plumbing issues to optimize customer experience with the product, and a set of dedicated resources to work on new features for customer acquisition. This requires that there is a roadmap for optimizing customer experience alongside enabling customer acquisition to plan for a balanced release cycle by delivering a core set per Sprint to enable both.

·         Do not copy competitor’s customer acquisition strategy.

This can be a death spiral when product leaders simply start copying competition.  While it’s crucial to learn from the competition, it’s even more important to look at your own strategy to acquire customers and delight them so they stay with you.  Here are the key probing questions to ask – Are you enabling outcomes that matter most to the customers and that value created by you is far greater than the competition? What can you do to raise the bar and delight the customers around key outcomes you are focused on? What outcomes are you enabling that competition cannot enable today? Are you adding more user friction than the competition? If so, what is your strategy to minimize that?  These are just a sample set of questions, which probably are pretty obvious.  It’s unfortunate that most product leaders and executive management teams simply gloss over the obvious and try to copy competition.

It is ideal to leverage strategy canvas (Refer: Blue Ocean Strategy – Authors: W. Chan Kim · Renée Mauborgne, and please see my blog from Sep 3-2013) to assess what outcomes matter most to the customers to develop your winning product strategy.  Products don’t simply become great by just copying!

·         Executive support.

Customer centric product life cycle management puts much greater burden on the executive management team so they can help and lead in managing customer and field organization’s expectations. They also need to ensure that they are not leading the product and engineering teams down the path which will result into incurring expensive technical debts on an ongoing basis.  There has to be a plan to pay that debt sooner rather than later, and Executive Management must support that.

A lack of executive support and poor understanding of the need to balance investment in customer acquisition and in optimizing customer experience can cost companies their fortune.  Temptation to acquire new customers often takes critical resources away from fixing the plumbing which becomes super expensive with each release and each new customer who migrates over to a new release. It also leads to engineering burn out as they spend long and non-productive hours in fixing all the problems they knew but they didn’t get the time or resources to fix. Among other things, this can also result into an exodus of top talent as no one wants to be wasting his/ her time fixing known issues!

At the end of the day, in order to have a sustainable growth and to have delighted customer experience, you need find the right balance between acquiring customers and optimizing customer experience!