Data Measurement and Analytics of B2B Products
As part of his role, a product manager makes numerous decisions daily. These decisions are made based on both experience-based intuition and data analysis. In the decision-making process, the product manager has to balance his/her sense of intuition by relying on the data s/he collects on the product. In general, the measured data is used to evaluate product performance. This promotes the understanding of whether the product development process heads in the right direction and whether the product matches the market needs.
Today, the field of analytics is well-developed and broad and offers a wide range of tools and means of examination. One of the significant challenges for the product manager is to characterize product measurement. The results of the product data measurement with a small number of users can vary dramatically depending on the analysis. Thus, performing a focused measurement on a particular customer set creates difficulty in completing the inference process. This is due to the fact that, in this situation, there is no large sample size that can help the product manager verify the measurement s/he has performed. This is just one example of the challenges the product manager faces on a daily basis; such challenges mostly concern data gathering and the ability to make decisions based on it.
The first part of the article will present an overview of practical tools for gathering information on B2B products with a small number of customers and B2C products at the beginning of their lifecycle. The second part of the article will present a way of handling the information about these products, focusing on the product metric characterization. Finally, the article will display a management method and an organizational work process regarding data gathering and product insights sharing.
Gathering Information
The product analysis process begins with collecting information and making measurements about the users’ usage of the product. At this stage, the main problem is a small amount of data and the inability to analyze it. Another problem is biased data and, as a result, biased insights of product usage. In this section, the article presents the information sources used by the product manager for product measurements.
If the number of product users is small, in a way that allows the product manager to hold a discussion with each of the users, then this is the initial effort s/he should focus on. However, if there is a large number of users, and the product manager cannot converse with all customers, measurement tools are required. The measurement tools presented below are not necessarily standard analytics tools.
Support Team — One of the main ways to gather data for measurements is information received from the support team. The product manager can collect data using the helpdesk tool, and can also understand what the typical customer complaints are, as well as the features that have most open tickets. With this information, the product manager can know whether there are problems in the product that needs to be addressed, what are the product’s processes that need to be improved, and which areas in the system inconvenience customers more than others. In practical terms, the support team sends an analysis report that contains the top five cases for the parsed day. When the product manager focuses on solving the problems that the support team receives, he allows users to be more independent and less reliant on the support team. As a result, there are fewer cases in which customers contact the company.
Another insight that this method can produce for the product manager is if the customer does not understand the feature or that the feature is too complicated; this is an education problem. In this situation, the customers can work with the feature, but they need the accompaniment of the professional service of the company. The product manager concludes that an effort should be put into the training on the use of the feature.
Another data collection element of the support team is of complaints about bugs in a specific feature. The data allows the product manager to analyze the status of the feature and to decide accordingly whether to resolve the bugs or reassess the existing problem more broadly. In practical terms, the product manager makes these deductions using an automated process that produces a ‘heatmap’ on the most common cases that emerge from the support team. When the product manager notices that a particular feature in the system is starting to “get on the radar” of the support team, s/he begins to check what the users have complained about.
Sales Team — Another way is to research the information received from the sales team. The sales team interacts with existing customers and potential customers at different phases of the product life. The data that the product manager can gather is a description of what are the blocks the sales team encounters in the product’s onboarding process while presenting the product’s demo and/or in the product’s POC with customers.
Integrators, Third-Party Products, APIs — Some tools allow the product manager to get data through integrators working with the product. The information that the product manager can obtain through researching the integrations of the product with external services is to characterize the source through which customers arrive and to carry out continuous monitoring. The ability to understand the channel through which the customer reaches the product allows us to define the user’s needs optimally, and as a result, to characterize the product accordingly.
Personal Experience within the Company — Another way to obtain information about the product is to offer stakeholders in the company to use it. The funnel’s personal experience can teach the product manager how the user works with the product and what are the user’s challenges.
Questioning — Another source is data gathered from an initiated survey among individual customers. The user has gained experience and formulated the perception of his personal experience while using the product. This opinion has excellent value in the data collection and research phase.
UI Analytics- A tool for tracking and collecting information automatically during the users’ activity.
In conclusion, there is a variety of sources that allow the product manager to gather information for research and analysis. On the one hand, we surveyed passive information-gathering tools, such as receiving data from the support team and receiving data from integrators connected to the product. On the other hand, we also presented practical information-gathering tools, such as collecting data from the sales team and getting opinions from stakeholders after a personal experience with the product, and proactive interviewing of customers.
Analyzing Information
The data collection and analysis process is based on the product’s metrics, and usually, 1–2 parameters are set. The meaningful metrics significantly affect the product when a change occurs; in most cases, it will be reflected by increasing the product’s usage and the number of users. To set a useful metric, the product manager needs to perform several preliminary actions. The first step is to characterize the purpose of the product and its end-goal. The second step is to define the product vision, and the third step is to identify the benefits and the unique value that the product provides to the customer.
It is relatively easy for early-stage products to set the metrics, but there is a problem of measuring information. The product has a few customers, and as a result, the measured metric is affected by many parameters. The challenge is to make measurements with small sample size. The primary way of coping with this challenge is to produce a variety of data sources for the product, as described at the beginning of the article. Another method of coping is to cross-reference information obtained from different sources, thus reducing the exposure of the measurement to the data volatility. This could be done, for example, by cross-referencing information through the UI, API measured data, and a third-party product interface. When the product manager cross-references the information obtained from these sources, he can analyze the product’s status.
Following is an example of a case of usage analysis in the product’s feature. The measured metric is the length of time it takes for a user to complete a feature funnel in a product. The measurement reveals that users work with a specific feature for several hours each day. In this situation, the product manager needs to determine if this is a useful statistic because it indicates that the feature is user-friendly and familiar. Alternatively, this measure is not right because it demonstrates that it is difficult for users to finish the funnel. The product manager crosses the data on the feature using data regarding duration with the support team. In addition, the product manager cross-references the information with the feedback s/he receives from the sales team. In this example, the product manager may find that the customer’s goals are to start the operation and finish it as quickly as possible. From this, s/he can conclude that the feature gives value to the user but needs to be improved. As the metric’s duration decreases, the customer gets the service s/he wants in a better way.
Sharing Information
This article presents one method out of the many available techniques; each product team can decide on an effective way of gathering information and presenting insights. The product manager will set a product monthly report event where the product team will present the new features in the system. The product manager will focus on introducing one or two functions in a meeting. In the session, the product manager presents the features’ performance and insights regarding the measurements. For example, what is used? Is the data useful or not? Should we continue in the same direction? Should an extension of the feature be added to the roadmap? At the monthly meeting, the product manager will compare compliance or non-compliance with the KPI. If a particular feature does not meet the KPI, a change is required, i.e., the team needs to decide whether to improve functionality or change it completely. Decision making in this session is the human process that is performed on top of the other processes related to the measurement tools.
The product manager produces a document describing the new features released, the amount of use, and critical insights of the information analysis. At the end of the meeting, s/he shares the information that was analyzed with the company stakeholders. Remember, each product team can decide how often to hold a discussion revolving around metrics — daily, weekly, monthly, quarterly, or annually. Furthermore, the mailing list of the summary may include only managers or only product/development/support/sales team.
The world, and as an extension, the market is very dynamic. The product manager has to do many experiments, examine the results quickly, and make quick decisions accordingly. To validate his/her insights, the product manager needs to be able to answer the following questions: Is the experiment I did beneficial? Is the feature heading in the proper direction? Does it match the market needs? The same method applies to metrics. If the product manager has performed a test on a particular parameter and understands that it does not drive the product towards the end-goal, then a new evaluation is required, along with defining which metric is relevant. The purpose of the metric is to advance the value of the product and the business.
In conclusion, the main goal of the product manager is to successfully use measurement tools and analytics tools and reach product insights. The main challenge of the product manager is to be attentive to the measurements that arise from product-use and, on the other hand, to be consistent with the customers’ needs to propose real value. This article offered a variety of practical tools for dealing with these challenges.
Written by Maayan Galperin
