Название: Delta Lake: Up and Running: Modern Data Lakehouse Architectures with Delta Lake (Final)
Автор: Bennie Haelen, Dan Davis
Издательство: O’Reilly Media, Inc.
Год: 2024
Страниц: 267
Язык: английский
Формат: pdf (true), epub (true)
Размер: 12.3 MB

With the surge in Big Data and AI, organizations can rapidly create data products. However, the effectiveness of their analytics and machine learning models depends on the data's quality. Delta Lake's open source format offers a robust lakehouse framework over platforms like Amazon S3, ADLS, and GCS.

This practical book shows data engineers, data scientists, and data analysts how to get Delta Lake and its features up and running. The ultimate goal of building data pipelines and applications is to gain insights from data. You'll understand how your storage solution choice determines the robustness and performance of the data pipeline, from raw data to insights.

First, we discuss why Delta Lake is an important tool for building modern enterprise data platforms and data science and AI solutions, followed by instructions on how to set up Delta Lake with Spark. Each of the subsequent chapters will walk you through the fundamental functions and operations of Delta Lake using step-by-step instructions and real-world examples.

The code examples in the book range from snippets that can be used in a PySpark shell to those designed to be run with a complete end-to-end notebook. In this book, all code snippets will be in Python, SQL, and, where necessary, shell commands.

You'll learn how to:
• Use modern data management and data engineering techniques
• Understand how ACID transactions bring reliability to data lakes at scale
• Run streaming and batch jobs against your data lake concurrently
• Execute update, delete, and merge commands against your data lake
• Use time travel to roll back and examine previous data versions
• Build a streaming data quality pipeline following the medallion architecture