This is a follow-up to the piece on the engagement model, in which we’ll discuss how the constructed model may aid in product knowledge, product development objectives, the search for insights, and the selection of the aha moment. All challenges will be accomplished using the example of an abstract marketplace for simplicity of comprehension; the vast majority of readers will be familiar with this sort of service, so it will be simple and clear.
The first part: https://medium.com/@sqweptic/product-analytics-engagement-model-22d53c96d169
Why does the model work?
First and foremost, ensure that the created model functions properly. It was previously assumed that the fundamental parameters had a close link (which expresses itself, among other things, in a high correlation), indicating that all parameters have comparable patterns. As a result, by emphasizing the segments according to the user model (for example, by splitting all users into deciles), the dynamics of all parameters in each segment may be traced, allowing the model to be tested and verified.
Because the fundamental parameters are a direct element of the model, it’s expected that all of them rise in sync, but all metrics, rise from a decile to a decile as well.
All other metrics, such as ARPU, retention, and lifetime (LT), are also increasing. As a consequence, the model accurately reflects the user’s engagement in the project, and because this is represented immediately on the cohorts.
What should the product’s goal be?
Before putting the engagement model into reality, it’s important considering how the distribution of users across the model might reveal the current status of the product.
Consider the following distribution variations to begin:
a) An idealistic product assumes that the most users are willing to learn about and use all of the product’s capabilities; that they successfully navigate the funnel, with no negative repercussions when problems emerge; and that users are satisfied after completing the intended activity. All of the metrics are quite high: the bounce rate (for paying clients) is close to 0%, the retention rate is close to 80%-100%, and the stickiness factor is near to 1. Discounts have minimal effect, and the newly developed feature affects all segments equally since users have formed a habit of using the product, making it virtually an essential one.
Due to the limited and continual fluctuation of the product range or content, regular revisions of pricing and marketing strategies, changes in external conditions, risks, and so on this degree of engagement is hardly feasible.
Perhaps one of the examples of such degree of engagement is Google Search which is an essential product for most internet users and users’ interacting with the product is quite the same over a period.
b) a saturated product – the majority of users think the product is good, a significant number of users are loyal, and the product’s structure and availability are obvious to the majority of users.
Metrics: exceptionally high conversion rate to targeted action, low bounce rate, user cohorts are doing well, and plateau is being formed despite a minor dip in the initial periods.
The link between the assortment of products and content, as well as the convenience of the offer, is the foundation for maintaining and attracting users (users understand and use the recommendations, search always leads to a variety of choices, even if there is no available product, the catalogue does not restrict the user and there are always product alternatives offered for the current request and for future). Marketing operations effectively attract new users (returning users do not respond since they are familiar with the product), the majority of whom engage with the product. Significant price offers are more successful for new users and less effective for returning users.
c) an optimized product – the product has a steady and significant user base, which expresses itself in specific limited segments (users are retained by certain features or offers), and consumers are willing to interact with other pages.
Metrics include the average conversion rate to desired action, the average bounce rate, and the number of people willing to return to the product.
At this point, a significant proportion of consumers is ready to interact with the product, and the effect begins to reveal itself more clearly in a/b experiments.
d) a conversional product – only a tiny percentage of users have developed a habit of using the product, the majority of users are uninterested in the product’s capabilities (or do not know how to utilize this feature), and the loyal audience is unstable and susceptible to external influences (activities of competitors, negative reviews).
Metrics: Low retention rate, average conversion rate to desired action, and high bounce rate. The product does not keep consumers effectively; marketing efforts and price offers are the most important factors in acquiring an audience.
Because only a tiny percentage of visitors visit child pages of the product, the impact of hypothesis testing using a / b testing outside the base flow is minimal, and a / b testing is used rarely. With a DAU of 100 thousand users, for example, a 2% gain in conversion from a child section visited by 1000 individuals is tough to reimburse in terms of administration and development expenditures. As a result, by boosting product engagement, increasing switches between sections/pages, and product engagement you expand the options for evaluating product hypotheses through experiments (a/b testing).
Prioritizing during product development
Let’s take a deeper look at how the engagement model enables you to determine what should be altered in the product. To do so, let us return to the target metric’s distribution in relation to the engagement model’s deciles. For example, this is LTV.
Assuming that we need to raise lifetime value by 10% of a product; according to the engagement model’s distribution, we look for quantiles that are similar in value; such LTV level has quantiles 67–70.
By separating these quantiles into an exact segment, you may characterize the metrics of this segment, that is, analyze how they engage with the product and compare this to the existing average values. Consider the following table of metrics for an abstract marketplace.
Users with a 10% uplift than the average are more likely to interact with search and recommendations. It makes sense to focus on the functionality/pages connected with the user’s transition to search and explanation of recommendations as product hypothesis, for example. In the case of an online store, this could be the functionality of the "similar search queries for a specific product’ block, which would be recommendations and would also transfer into search, or other recommendation algorithms, such as taking data on user search queries when building a vector of recommendations for a product/user.
As a result, employing the engagement model makes it simple to determine which product development path should be prioritized in order to achieve the metric’s intended value.
The capacity of product owners, managers, and analysts to establish causal links for metric changes is critical in this scenario (as in the table above).
With such an analysis, it should be borne in mind that if we compare very different segments, for example, churned and returning ones, we will compare radically different segments, which will not give exact points in the product that are worth improving.
Metrics’ elasticity. Financial and product indicators and their relationship
Do you know how much LTV will change if the bounce rate is reduced by 5%?
If a user does not fall within the bounce rate, they will most likely go through the payment funnel (CTR), and if they go through the payment funnel, it will most likely be turned into a target action (CR). As a consequence, the user will have the following binominal parameters: bounce – 0, ctr – 1, cr – 1. The user has these criteria because he is driven and eager to utilize the product since they are satisfied with it. It is feasible to identify comparable users based on their propensity to be engaged in the product using the engagement model; such a segment of similar users will have a particular probability in each action and a certain connection between metrics.
By selecting a significant number of such segments, you can obtain the relationship of metrics throughout the entire product, which is quite easy – customers in each n-quantile will already be classified as comparable users in terms of engagement.
Let’s take two metrics as an example: ARPU and conversion rate (CR) to purchase. Using these metrics we will collect statistics of the date and quantile of the engagement model with the following structure:
It makes sense to alter the measurements daily / weekly because the metrics fluctuate over time (seasonality – winter/summer, marketing activity, and other reasons). It’s also important to double-check that the quantile segment is suitably representative and that the metrics inside the segment have relatively small variations.
After receiving such information, you may create a graph depicting the link between metrics.
The chart depicts the relationship between metrics over time (collected for several months); as a consequence, you may use linear regression, to approximate this relationship and obtain an estimated value for every change in the metric:
After receiving the final ARPU, you may calculate the metrics’ elasticity:
As a result, the elasticity of all indicators in a product, as well as their relationship with financial and business metrics, can be assessed, allowing such a tool to be used for the financial assessment of product hypotheses and highlighting priority in product hypotheses execution. Linear regression may be created not only on the basis of the relationship of two metrics, but also on the relationship of many metrics, such as CR/CTR and ARPU. The utilization of the correlation between product and business indicators may also be included into unit-economy calculations.
Of course, because it is based solely on the interconnection of processes inside the system, such a tool is more closely associated with simulation modeling, is subject to biases, and does not completely reflect cause-and-effect relationships.
Keeping track of a product’s attractiveness and seeking insights
Using the engagement model, it is also feasible to track how the user’s interaction with the product evolves over time, in addition to being able to identify specific users who interact with the product. As with any cohort, some consumers begin to use the product less actively over time, while others begin to use it more actively.
By dividing users with three segments of engagement relatively the distribution of the model: high (the most engaged users), middle and low (the least engaged users).
The most engages users have a higher probability of staying in the product over time. Those less involved are more likely to end up in the churn.
As a result, there exist two dynamics of user state: one in which the user began to take less activities in comparison to previous visits, lowering his involvement in contrast to other users (red arrows), and the other in which the user increased his activities (green arrows).
An analysis of users who have increased their engagement with the product today compared to yesterday, allows us to determine what factors, such as engagement with which functionality or section of the product, influence the Growth of metrics, which leads to the formation of a habit of using the product.
Since within such analytics it is possible to form the exact segment of users who began to interact more with the product, to research the activity and actions of this segment in the product that affected to the return to the product and the increase in engagement.
On the other hand, users who continue to interact with the product, albeit less actively (red arrows), are susceptible to in-product retention methods through the ui, instead of mailing/push notifications/alerts that mostly is focused on churned users.
Other uses
The engagement model is a tool that may be used in a variety of situations.
The model can demonstrate the impact of the product hypothesis on the product in various engagement segments. Impact values, for example, may be assessed on a subset of highly engaged users who are more devoted to the interface and better equipped to engage with it. Alternatively, to assess how consumers who are uninterested in the product respond to the new functionality.
The model also allows you to compare similar users who did and did not perform the same action. For example, if you take the same segment by engagement with the product and compare those who added to the wishlist to those who did not, you can say more precisely what effect this functionality has on one or more metrics. This opportunity is obtained due to the fact that the interaction with the product is compared with users similar in their characteristics (more representative groups are compared) and, as a result, lowering the bias.
Dynamic of the metrics of highly engaged users, for example, can become a quantitative counterpart of NPS analysis, allowing to more rapidly follow and examine the behaviour of the most devoted users.
In order to understand user behaviour in the product, usage patterns, characteristics and manners of interactions, it is necessary to perform a variety of research, however, one kind of research can be done using the tool described in this article.
Of course, much more information can be given about the engagement model. In the article, I want to simplify many points and tell more about the results that can be obtained by building the model, so perhaps the article turned out to be confusing in some sections.
