Название: Automating Data Transformations: Enable Your Organization to Solve the Largest Bottleneck in Analytics
Автор: Satish Jayanthi, Armon Petrossian
Издательство: O’Reilly Media, Inc.
Год: 2023-04-25
Язык: английский
Формат: pdf, epub (true), mobi
Размер: 10.2 MB

The modern data stack has evolved rapidly in the past decade. Yet, as enterprises migrate vast amounts of data from on-premises platforms to the cloud, data teams continue to face limitations executing data transformation at scale. Data transformation is an integral part of the analytics workflow–but it's also the most time-consuming, expensive, and error-prone part of the process.

Many have heard the term data velocity, popularized by Oscar Herencia as a part of his five V’s of dаta: volume, velocity, variety, veracity, and value. For the past two decades, the first four V’s have grown exponentially, but what about the most important—value? Data transformation exists to deliver value and drive tangible improvements in business outcomes. While many organizations have built data warehouses and lakes, growing the volume of their information, how many have seen their data’s value grow proportionally?

Today, integrating data into business operations is table stakes. Winning organizations will have the most performant and scalable architectures, prioritize the value of their outputs, and place the greatest emphasis on results. We’ve seen this pattern in other operational groups: engineering, sales, and marketing, to name a few. From workplace wikis to a slew of Slack apps, automation and tooling drive process improvement, which multiplies the value-add of passionate, curious employees—this is the premise of the modern data stack (MDS).

The MDS is the practitioner’s toolbelt. Under its umbrella are products for data ingestion, storage, transformation, analytics, and governance. These solutions enable data teams to be lean and efficient: only a few years ago most were built from scratch, limiting robust data analysis to organizations with a fleet of data engineers and architects. As a result, those who leverage the MDS are capable of delivering more value at a more rapid pace.

What You Will Learn:
From this report, you’ll gain:

• A conceptual understanding of the MDS
• Context on where current tooling shines and where it falls short
• A vision for the future of data transformation: how we see transformation tooling evolving
• A framework for automating the transformation layer
• A path to a scalable, robust data system that creates business value

Who This Report Is For:
Our intention is to deliver value to data stakeholders: directors of analytics, data leaders, and CDOs/CTOs. That being said, we feel that a diverse group of data and product practitioners will benefit from this report. A successful tech stack takes a village—not just the data team. The DevOps, infrastructure, and product teams are all working to pull data initiatives forward, ideally from the same end of the rope. Here’s a brief overview of what various groups might gain from our discussion:

• Analytics and data leaders will learn more about the MDS and current industry trends. This will improve decision making and provide relevant context on services, frameworks, and build/buy decisions in the data space.

• Data/analytics engineers will gain a better understanding of why a particular technology makes sense in the context of their organization. Pondering high-level decisions can be a good exercise for individual contributors and a window into strategic planning. We challenge our readers in this group to think critically about their own infrastructure and how it fits into the MDS.

• DevOps/infrastructure engineers are often some of the closest partners with the data team. Because of the tight connection between cloud services and the MDS, these engineers are frequently deploying infrastructure at the behest of data stakeholders. Understanding the MDS can lead to empathy for, and insight into, the processes of data teams.

• Data architects are experts in the patterns and architecture of our data. While they know best how to structure data systems, their design choices rely heavily on downstream users (data and product teams). From this report, data architects will be enabled to make informed decisions and consider the perspective of their partners on the data team.

Contents: