In this article I share the experience of my first three months at Stuart as a Data Product Manager for our internal data platform; what it means to sit at this powerful intersection between data and product; and why it’s critical for a modern data platform team like ours to “think product” in truly delivering business value. I’ll draw on the challenges faced when looking at a data platform through a product lens, redefining it from simply a “tech stack” to what it really is: a product that delivers value to hundreds of business users day in day out. My hope is that this will serve as a guide to data platform teams at any stage of becoming more product-led while contributing to the emerging space of data product more generally.
Over recent years most companies have adopted a data platform as a way to manage how data is ingested, stored, transformed and accessed at scale. The end goal is typically to support the business in making fast and reliable data-driven decisions, and here at Stuart we’re no different.
In the face of exploding volume and complexity of data which needs no further explanation, the focus of data platform teams have understandably been concentrated on stability and reliability at scale. For example, ensuring the smooth running of business-critical pipelines, minimising downtime or optimising query efficiency.
While companies large and small have made considerable gains in building a scalable and sustainable architecture, we’re left with the uncomfortable questions: is what we’re doing truly providing value? Do we really know who our users are and understand their needs? If so, can they generate insights in a fast and reliable way? As long as users don’t complain and pipelines don’t fail, does that mean all is well? For all our investment in data, are we seeing the return?
These questions that touch on the concept of value are particularly difficult for an internal platform that doesn’t sell to or make money from our users. Moreover, the added complexity of quantifying the value that data brings forces us to consider alternative definitions of value:
While the importance of the former cannot be underestimated and should play a critical part in goal setting and prioritisation, this is often a question of modifying process. Meanwhile, the latter is a more complex topic, and especially so for a data platform intended for business (i.e. human) decision-making. To better understand users requires more than a change in process — we need a change of mindset.
When the data platform is viewed purely from a technical lens, we put ourselves at risk of forgetting our users and their needs, building solutions before we’ve understood the problem and further increasing the data/value gap.
These questions are particularly important to ask now, when the growing discussion around data mesh suggests we’re on the verge of a radical shift in how business interacts with and governs data. This only further increases the need to proactively and continuously understand our users and their evolving needs.
Much in the same way that experimentation platforms, algorithms, and even the data itself are being increasingly considered products, we turn our attention to the data platform. For simplicity, consider the data platform to refer to all the tech and tools that collectively serve the business to make data-driven decisions.
If the previous section wasn’t enough to convince us, there is one concept that helps sum up the need for product thinking for a data platform more than anything:
Accessing and working with data is a user experience.
In the same way we use applications for almost everything in our personal lives from measuring our heart rates to finding a new house, the experience of accessing data to make decisions should be no different. But the concept of UX is rarely heard among data teams. This is hardly surprising if the data platform is viewed entirely as a technical problem and resourced accordingly; but perhaps added to this is the idea that as user-facing data tools are frequently bought and often come with a UI, that these UIs automatically “take care” of our user needs, i.e our job is done.
But a good UI cannot replace the need for a good UX. To think that in most modern organisations, hundreds or even thousands of users are interacting with data on a daily basis, the idea of not continuously investing in discovering users’ needs, pain points and desires, of not using this user context to shape our roadmap for success, in other words, not thinking of our data platform as a product should concern anyone investing in it in the first place.
Here at Stuart the Data team is fortunate to sit alongside a strong Product team whose philosophy is centred on continuous discovery: identifying opportunities in the form of user needs, pain points and desires and using this to guide our vision, strategy and goals. This means that as well as building products that make sense both from a technical standpoint and aligned with company goals, we’re also strongly led by our users. This approach means we’re more likely to deliver a data platform that users love, and less likely to jump into solutions that ultimately only we in the platform teams love.
Until now we’ve mostly discussed the why. It’s now time to move from theory to reality and put this into practice: the how. The complex make-up of the data platform doesn’t make it easy to define as a product by any means. Here we present our “first principles” approach to this, which focused on three broad areas:
Together with other data leads and interviews with key business stakeholders we investigated each of these areas one by one. To ensure it didn’t become a one-off exercise we made them as visible as possible internally, e.g as part of onboarding a new Data or BI Engineer. Doing this also incentivises us to keep them relevant and up to date.
If we want to shift to a more user-centric approach, understanding who our different users are feels like a logical starting point.
Why would anyone make use of a data platform in the first place? What are the key actions our combined user personas should expect to be able to perform autonomously? These were the questions we asked during this step.
With a better overview of the users and motives for using a data platform, we turn our focus inwards and ask what our data platform offers. Similar to how market-facing products sit on a shelf or under the “Our Products” dropdown on a website, how do we represent our data platform as a portfolio of products?
One of the advantages of dividing our platform in this way is that we can build a set of KPIs for each product component, which would then form the basis of future OKRs:
At a later stage, we mapped the more familiar user interfaces (usually third party tools) to each component, so that users can clearly navigate to the specific tool(s) for accessing each component. We felt it important to distinguish between components and interfaces for these reasons:
By taking a step back and linking the use cases, user personas, components this enables us to:
We’ve introduced the concept of a data platform as a product, moving beyond simply viewing it as a tech stack. We did this by identifying who our users are, their intents and what our product offering is that can meet those needs.
As we’ve witnessed at Stuart most modern data platforms are complex and multifaceted, so the simple act of putting these on paper and visualising our platform as a collection of products that deliver value to our users is already a meaningful first step.
Our work is by no means done here — we are only at the start of our journey to becoming more product-led. These practices and principles will form a solid foundation for what comes next: discovering opportunities on an ongoing basis. Opportunities that will play a key role in building our strategy and OKRs for the coming year, becoming a data platform that continuously brings value to our users and to our business.
Main reading sources: