In this post, I aim to give you all a complete understanding of product instrumentation but before we discuss it let us talk about one of the most common questions that every product manager has to answer about his/her product.
“Okay so you have built some great features but how do you know that are helping the customers?”
The answer to this question is simple for most product managers. They would give out a list of metrics to measure the success of their feature/product.
We are living in a world where data plays the most important role in decision making. It is no longer limited to a few important metrics that you track for your product. We are now empowered with the most advanced product analytics and engagement solutions that help us track the smallest interaction in the product.
There is although one problem that most of us face while adapting to this new world of so much data surrounding us.
“My product has thousands of actions that a user can potentially take, how do I decide what needs to be tracked?”
Engagement and Analytics solutions can help you get answers to all your questions related to usage of your product but you still need to map your entire product so that you can get answers to all your questions in the easiest manner. This process of mapping the product is called Product Instrumentation.
Let me try to explain this with a real life example of how we implemented it long back.
When I started out as a product manager I used to face this challenge of managing internal stakeholders. Despite of doing a lot of customer and domain research , I was constantly asked to give more data to support my hypotheses of building features and every time we used to release a feature I was always asked for adoption numbers. It was that time we realized that we need to start measuring all the actions that a user can take on the app.
Challenges
- We had 3 products in our platform and we had no clue where to start.
- We did not know if there was a tool out there that could help us solve this problem.
- We realized that in order to track everything it would require a lot of developer dependency. We had a small team of developers and getting resources for setting up events was a major problem for us.
Solution
This is how we approached these challenges
- Dividing the entire product into modules: The first step to making the entire process easy was dividing multiple products into modules, components, and features. This helped us focus on a particular area of the product to map it out completely
- Questions that you need answers for: Once we had the entire product broken down into focus areas we started to list down all the potential questions that we could need answers for immediately or at a later time.
This was one of the most important steps to ensure that we are holistically mapping our entire product with relevant tracking metrics.
- Metric Nomenclature: Thinking about giving events a name was not the hardest part at the beginning and thinking from first principles we decided to use this nomenclature
"Component - Action" For example: Project - Created
Note: A lot of people make this mistake of not taking nomenclature seriously but trust me once you have hundreds of metrics it will become a nightmare for you to navigate through these metrics if you don’t follow a strict nomenclature.
- Mapping the metrics to its source: Once we knew the questions that we needed answers for we decided to find the data sources which could help us get answers to those questions. The first place we looked at was the product database. For instance in one of the products called ‘VWO FullStack” the most important question for us was “Number of projects created” the source of data for this information was our product database.
Similarly, there are certain data points that are not directly accessible from a product database. It could be interaction with radio buttons, dropdowns, etc where the value keeps updating. For such data points, you can use client-side event tracking systems such as Segment
Note: This step of mapping the events to the data source is not mandatory but it helps you keep a track of the source of the data point/event.
- Properties associated with every data point/event: Next step for our was the extra information we need for every data point. For instance as mentioned above of one the important question was “Number of projects created” we need more information along with this event.
Event Name : Project Created
Properties: Basic (identify) property and Event specific property
Examples of basic properties: These properties are common across all the tracked events/data points. For eg. Account ID , User ID , Plan_type , MRR , LTV, etc.
Examples of event-specific property: Let us say that a project can be created from 2-3 different locations within the application such as a dashboard or projects list screen. In this case, instead of sending 2 separate events we would add a property called “Location” which would have 2 values i.e dashboard, project lists. This will help us in the analysis as well as reducing unnecessary events.
- Ensuring that we keep mapping all the new feature: To make this a part of our feature development process we made it a necessary part of our specification document i.e Every time a product owner builds a specification document he/she has to map all the questions that they would need answers for and setup tracking of the data points.
Summarizing important points to keep in mind
- It might seem a lot of overhead when you start implementing these data points as it requires developers but nowadays there are marketer-friendly solutions available.
- There are products such as Gainsight PX that help you map your entire product without using excel to maintain a list.
- Mapping the entire product thoroughly with proper nomenclature goes a long way in your analytics game!
I hope you were able to understand what product instrumentation means. There are many ways you can do it right but the more important part is to keep improving. I am hoping to learn and evolve this system and would love to hear your thoughts in the comments below.
In the next post I will talk about how we leveraged all the information to build a data driven culture in our organization.
Fun fact: We started working on this three years back and we had no clue that it was called “Product Instrumentation” 😀 Moral of the story is that it is not important to focus on the ‘jargon’ but to really understand your problems and ways in which you can solve it.
