Having worked in data science for almost a decade, spanning diverse domains like business analytics, business intelligence, advanced analytics, machine learning, and artificial intelligence, I've come to a sobering realization: unlocking value from data remains a perpetual challenge. While my early career was spent crafting hands-on data science solutions, more recently I co-founded Mindfuel and began to study and analyze from a more strategic perspective the organizational ecosystems where data initiatives are born. Here's what I have found out so far.
The data science surge of the 2010s
The data landscape of the 2010s was vastly different from today's. A decade ago, the European market was dominated by business intelligence solutions, with data typically stored in relational databases and manipulated using SQL programming language. The culmination of a day's work often resulted in the generation of business reports or the deployment of dashboards accessible to business users.
However, the winds of change were blowing across the Atlantic from the United States, carrying with them a new phenomenon known as "data science”. It was the beginning of a hype. People wanted to use advanced methods to extract more value from their data—to uncover deeper insights, glean more information, and ultimately, take better actions.
This demand gave rise to a wave of innovation. We analysts learned to develop models, create showcases, build prototypes and even deploy first pilots. Our efforts bore fruit as we proudly showcased our company's latest projects to executives, who greeted them with a mix of excitement and anticipation. Moreover, they provided us with further financial funding and budgets to keep going, with clear expectations: Please, deliver value from these initiatives.
Some of these stakeholders made substantial investments, pouring funds into centers of excellence, data laboratories, AI hubs, and data divisions, alongside procuring the requisite technologies to construct a cutting-edge 'data universe'. Chief Data Officers (CDOs) were hired to dig the gold buried within the data landscape. The market was growing limitlessly, fueled by optimism and boundless potential. However, the tide turned abruptly when the first stakeholder asked a magical question: "When can I expect to see returns on my investment?"
This question marked a turning point in the narrative, signaling a shift in expectations. It became increasingly clear that merely developing technical solutions was insufficient to deliver the promised value.
As reality set in, the euphoria surrounding data initiatives began to wane. It became apparent that only a fraction of these endeavors were truly yielding returns on investment, with many falling short of expectations. Research institutions intervened, shedding light on the stark reality that ~90% of all data initiatives won’t make it into production. This sobering realization led to a recalibration of priorities. Finally, the hype flattened slowly towards the end of the decade and today’s decision-makers demand concrete ROI concepts before allocating budgets and investments, signaling a shift from the era of blind optimism towards a more pragmatic approach to data-driven initiatives.
Why we have failed so far
Let's face it—we've fallen short. It's not solely the technology's fault; we've neglected the crucial organizational aspect. Over the past decade, significant strides have been made in developing superior models, refining infrastructures, and enhancing platforms — or to sum it up: better solutions. Thanks to the efforts of both established players and novel open-source communities, executing a regression model or scaling up computational power is now as easy as a simple click.
Yet, for some reason, the necessary changes to the organizational environment were overlooked. Here are the key observations I've made:
- Despite claiming to be agile, many people struggle to transition from traditional waterfall methodologies to agile approaches effectively.
- Organizations still lack a well-defined framework for handling data product development and seamless integration within their respective ecosystems.
- The data world often disregards user needs and preferences, placing too much focus on feasibility.
- Fostering collaboration among business units, particularly concerning data, presents significant cultural and mindset challenges.
- A common tendency to focus on output over outcome, neglecting the importance of holistic product development.
- Accountability for the business case and securing a good return on investment remains unclear.
These observations extend beyond the typical challenges within data science such as how much data is accessible, available, or ready to be analyzed but go far beyond the data space.
Learnings from digital product management
Frustrated with the existing status quo, in 2019, I shared my insights with Max, a close friend and eventual co-founder at Mindfuel, who happened to be working as a digital product manager at the time. His response was eye-opening, as he highlighted several fundamental flaws in our approach:
- Developing projects rather than products.
- Ignoring desirability in solution development.
- Overlooking marketability, despite the competitive landscape.
- The absence of clear ownership of solution viability.
His words sparked a whirlwind of emotions within me, from frustration to outright rage, eventually culminating in a newfound sense of motivation. Max's astute observations and the way he described the core of product management in greater detail, and the role and responsibilities of a product manager specifically, raised critical questions regarding the applicability of traditional product management principles to the realm of data science:
What does this all mean for a data science project? Could a product owner or project lead assume responsibility for the ROI of a data science project? Alternatively, should this responsibility lie with the typical business owner, who provides requirements and domain knowledge? Yet, does this individual truly possess the requisite ownership of the solution, or are they merely end-users upon its completion? These questions indicated the absence of a defined product manager role within the data space—a noticeable omission that motivated further examination.
With a strong commitment to delivering on our promise of value creation to stakeholders, Max and I embarked on the next phase of our journey: Data Product Management. Adapting methodologies and frameworks from traditional (digital) product management, we opened up a fresh perspective on how data initiatives could be realized and delivered moving forward. By blending principles from agile frameworks, execution engines, and product management tools with artifacts from CRISP-DM, ML- & DataOps, and insights from the data strategy domain, we crafted a holistic approach greater than the sum of its individual components.
Why the world needs Datapreneurs
However, all our efforts led to the fact that there may be more to add to this story than just data product management. We've come to recognize that data product management is merely part of a larger discipline known as Datapreneurship.
A Datapreneur always truly cares about the current and future use of data whilst considering the product’s greater cause. A Datapreneur delights the world with innovative and sustainable data products people love. Every products/he designs and builds makes its environment better – step by step, idea by idea, product by product.
The Datapreneur is a transformative figure—a modern-day equivalent of the pioneering data scientist from a decade ago. Acting as the captain of each initiative, they steer the ship through tumultuous waters, navigating risks while relentlessly pursuing the promised return on investment. Beyond merely coordinating machine learning models, the Datapreneur envisions a product's greater purpose, always mindful of user preferences, creating business-critical success accordingly whilst understanding and tackling the challenges of the data world in practice.
They possess the resilience to embrace failure, leveraging each setback as a springboard for growth. Moreover, they defend ethical principles, recognizing the responsibility inherent in managing investor funds and safeguarding against the potential misuse of data. A Datapreneur always truly cares how data is used in a product today — and will be used in the future — while considering the product’s greater cause.
Yet, the Datapreneur's role transcends individual products—it encompasses a commitment to fostering innovation and driving value creation. They are custodians of data integrity, ensuring that each product not only meets current needs but also anticipates future demands. With each innovative and sustainable data product they deliver, the Datapreneur leaves an indelible mark on the world, shaping a better future.
In envisioning a world where Datapreneurship is part of every data initiative, we remain steadfast in our mission to reduce the failure rate of data initiatives. We welcome fellow travelers on this journey, eager to collaborate and forge a path towards digital success. Join our Datapreneurs Club - a dynamic community for individuals passionate about using data to drive business growth.
If you want to find out more about Data Product Management, download our Ultimate Guide.
