Inconsistency over metrics is a painful issue. Metrics and KPI's are vital to help stakeholders track performance and align towards goals. In most organizations, metric definitions are spread across spreadsheets, internal dashboards, and data tools. Reporting supposedly simple metrics such as revenue can become very tricky due to multiple definitions and lack of trust in the data. This fragmented system leads to team disagreements about definitions, ownership, and accuracy of metrics. A lot of manual work is required to settle disagreements and align stakeholders.
This article summarises insights gathered during a virtual discussion involving
**Allegra Holland, GTM strategist @Transform. **Transform is a metrics store amazing at creating companywide alignment around key metrics through its tool. The organization’s founding team met at Airbnb. They decided to launch Transform after witnessing how Airbnb’s internal metrics store transformed the way teams made decisions and leveraged metrics.
Zdenek, product manager @Gooddata. GoodData is an embedded BI and analytics platform that seeks to help organizations build analytics for their end-users. GoodData provides self-service analytics, low-code/no-code interfaces, and embeddable data visualization.
Burak, founder @MetriQL. MetriQL is an open-source project that lets you define your metrics as code in a central metric store. MetriQL was acquired by Liveramp, a platform for data connectivity that seeks to enable companies to better connect, control, and activate data to improve customer experience.
Together with these three experts, we discussed the origin of the metrics inconsistency issue, what a successful solution should look like, and the role of metrics stores in solving the problem.
The roots of inconsistency over metrics
Cloud data warehouses provide a unique source of truth for data, making the metrics consistency issue less worrisome to some extent, as people are all working with the same data. However, the data democratization trend means that data is consumed by many more people now, and we’ve also witnessed an explosion of tooling in the modern data stack. There are too many tools, using too many languages to define the metrics. This creates complete chaos. This is only going to get worse in the future as tools continue to multiply, and more people start using data.
Zdenek: The main issue is consistency over metrics. In businesses, where you try to track the performance of a metric, such as revenue, you get a lot of different answers from different departments. Metrics stores were born out of the need to align all the customers to see the same numbers and be aligned on business decisions.
Burak: Metrics stores emerged following the growing complexity of the data infrastructure. Let’s take the example of revenue. Historically, things were easier because the revenue stream was coming from a single channel. Right now, businesses contact customers through a lot of different channels. So to be able to calculate revenue, we need to gather data from many different applications. It’s not easy to build up metrics in this context. You need to take into account the multiple downstream data sources and store all these data in a single place to be able to calculate it. That’s why we have the data warehouse. The second reason is: you want to be consuming metrics from multiple tools. That’s the whole idea of headless BI. Now, in a single organization, there are different persona who need to interact with these metrics. All these departments prefer different tools. We’re going in this decentralization era in the B2B tools as well because we want to use the right tools for the right job. All these tools should have access to the metrics in a convenient way. That’s why we have the metrics stores, where you can define everything at once and then talk to the downstream tools for different use cases.
Zdenek: The main use case for the metrics store is metrics consumption. People in the organization need to be able to consume metrics from a consistent place and rely on the fact that the metric is defined consistently itself. By, this, we mean the expression computing the metric is the same everywhere. Another use case is discovery. A metrics store makes it easier to find the metrics and the documentation around them. It makes it possible for people to drill down how these metrics were computed exactly. Finally, metrics stores are useful for change management. If something changes in the way a metric is computed, people need to be notified because it will affect their work.
Allegra: Key KPIs drive organizational alignment across the board. An important use case for the MS is that it allows specific departments to share the metrics that other departments may care about and that everyone should align on. There’s also an exploratory use case. Metrics stores help people understand the business by understanding the metrics which the business cares about. When someone recently joined a company, it’s useful for them to understand how their company thinks about themselves. To this end, it’s interesting to do some exploratory understanding of what metrics are and how they extend to different operational workflows. It’s the one-stop-shop to see how the business thinks about itself and how performance changes over time.
Consistency: metrics stores bring about a consensus on how metrics are defined and thus more alignment across the board.
Productivity: When a metric is properly quantified, it can simply be filtered out by different criteria so that analysts don’t have to write another SQL query but just switch a “where” clause to pull in one other field. This allows analysts to offload the work to business people who care about specific business use cases. Now the analyst has time freed up to spend time on more value-generating analysis rather than just re-write SQL queries.
The metrics store is essentially the layer between the database and the data consumption tools. For the moment, we mostly used dbt to define our metrics and use the downstream tools (BI tools) to analyze them. But the downstream tools just see database tables. The metrics store is the layer between the transformation tool and the downstream tools, allowing us to expose all these metrics to downstream tools in a different form than just tables and columns. It allows presenting metrics in terms of numbers, which are much more consumable than queries. That’s because queries are still foreign to quite a lot of people, while numbers are much more familiar.
Burak: Data catalogs are aimed at people that know how databases/ columns work. The metrics stores allow the data catalog to be elevated one level up by making metrics consumable by regular people, who don’t know about data structures.
Allegra: Data catalogs are sort of the raw data assets in the development environment for data where technical persons can go and understand the lineage of data, where a piece of data comes from, etc.. Exposing that lineage and passing that through to the metrics store allows end-users to feel more comfortable about how these metrics are defined. The metrics store is somehow the productionalization of the data that gets served to the business so that the business can make decisions based on that data.
A lot of different personas use the metrics store. Technical folks define metrics, providing the necessary context around them to ensure trust in how those metrics are going to be used downstream. The work is then offloaded to business users, who can go to the metrics store and search the metrics catalog to find particular metrics and filter them for their particular business use cases. Metrics stores reconcile business and technical users, with technical users defining and giving context around metrics while business users consume them. they consume without altering the definition for their business reporting needs.
We write about all the processes involved when leveraging data assets: from the modern data stack to data teams composition, to data governance. Our blog covers the technical and the less technical aspects of creating tangible value from data.
At Castor, we are building a data documentation tool for the Notion, Figma, Slack generation. We designed our catalog software to be easy to use, delightful and friendly.
Want to check it out? Reach out to us and we will show you a demo.