O’Reilly Insights
Tech Should Take a Back Seat for Data Project Success
Understanding business context is more important than knowing the cutting-edge tools
Editor’s Note: Co-founder and CTO of Lenses.io, Andrew Stevenson, shares what he has learned about the critical importance of understanding business contexts for those working in technology.
Data has always been a constant in my career.
From the beginning as a C++ developer, to my switch to data engineering, my experience managing increasing velocity and volumes of data — for use cases such as high-frequency trading — forced me to lean on cutting-edge Open Source technologies.
I was one of the fortunate ones. I worked with an elite set of handsomely-paid developers, at the peak of the Big Data revolution.
Cutting-edge, as we were, managing the technology, wasn’t our biggest challenge. Our top challenge was understanding the business context and what to do with the technology.
As a dev, I shouldn’t be expected to configure complex infrastructure, build and operate pipelines, and also be an expert in credit market risk.
Time and again, I witnessed business analysts sidelined in favor of data engineers, who spent all day battling Open Source technologies with little or no context of the data or the intended outcome.
The technology-first focus often led to Resume++ where technologists were empowered and enamored with technology rather than focusing on business objectives.
The result was slow progress, project failures and budget overruns. This especially hurt businesses during the 2008 crash and we’re seeing that impact again in 2020 with COVID.
The greatest success and innovation I’ve witnessed is when end users (business users, data analysts, data scientists) are given the correct tooling and the access to explore, process, and operate data themselves.
Modern data platforms are distributed, generally Open Source, and their tooling, if any, isn’t enterprise-ready. Therefore, organizations must turn to hard-to-find and highly-skilled engineers. Unfortunately, these engineers have a limited understanding of the data and the business context.
Some organizations with “elite” R&D will develop self-service tools at a huge expense. For those without the “elite” resources, these technologies are much less accessible. This exacerbates the divide between the web-scale companies (Google, Spotify, Uber, Amazon etc.) and the rest of the industry.
Technology should be an enabler to deliver data products.
To achieve this, best-of-breed technologies should be available in a “data mesh” type architecture with a decoupled self-service and governance layer. This allows end-users to apply DataOps practices to operate data and applications without infrastructure knowledge. Workloads should include data cataloging, RBAC controls, data masking, and the ability to create virtual workspaces that allow teams to explore, process, and move data through low-code applications.
Cloud providers have created “technology intensity” offering managed solutions for Apache Kafka such as AWS MSK and Azure HDInsight and Kubernetes. So, the role of data engineers should shift away from building pipelines and operating data infrastructure, to focus on delivering self-service tooling that allows end-users to operate data themselves and build “data intensity.”
My co-founders and I created Lenses.io to help every organization transition to the new data economy. Our technology gives businesses confidence through increased visibility and governance to adopt powerful data technologies such as Apache Kafka and Kubernetes — so users at any skill level are able to deliver real-time data experiences.
Learn faster. Dig deeper. See farther.
Join the O’Reilly online learning platform. Get a free trial today and find answers on the fly, or master something new and useful.
O’Reilly Media on Medium is a space curated by O’Reilly editors, offering advice, insights, viewpoints and experiences on tech.
Andrew Stevenson is the Chief Technology Officer and co-founder of Lenses.io. He leads Lenses.io’s world-class engineering team and technical strategy. With more than 20 years of experience with real time data, Andrew has led and architected big data projects for banking, retail and energy sector companies including Eneco, Barclays, ING and IMC, and is a leading Open Source contributor.
