Название: Kafka Connect: Build and Run Data Pipelines (12th Early Release)
Автор: Mickael Maison, Kate Stanley
Издательство: O’Reilly Media, Inc.
Год: 2023-08-23
Страниц: 512
Язык: английский
Формат: pdf, mobi, epub (true)
Размер: 10.9 MB

Used by more than 80% of Fortune 100 companies, Apache Kafka has become the de facto event streaming platform. Kafka Connect is a key component of Kafka that lets you flow data between your existing systems and Kafka to process data in real time.

With this practical guide, authors Mickael Maison and Kate Stanley show data engineers, site reliability engineers, and application developers how to build data pipelines between Kafka clusters and a variety of data sources and sinks. Connect allows you to quickly adopt Kafka by tapping into existing data and enabling many advanced use cases. No matter where you are in your event streaming journey, Kafka Connect is the ideal tool for building a modern data pipeline.

Consensus protocols, stream processing, distributed systems—in the midst of all the exciting ideas in the streaming world it can be easy to overlook the role of the humble connector. But actually connectors solve the most fundamental problem in the streaming world—in a world of data at rest, how can you get any streams at all? How can you plug your data streaming platform into the rest of the business?

Kafka Connect was designed to make that easier. Before the Kafka Connect framework existed we saw people building integrations with Apache Kafka and repeating the same mistakes. Reading data from one system and writing it to another seems simple enough, but there can be a lot of hidden complexity. What happens if a machine fails? What happens when requests time out? How can you scale up your integration? It seemed like each Kafka integration had to solve these problems from scratch. Kafka Connect was designed to separate out the logic of reading and writing to a particular system from a general framework for building, operating, and scaling these integrations. And indeed, the simplicity of the idea hides a lot of important ways that Kafka Connect is different from other integration or connector layers.

Learn Connect's capabilities, main concepts, and terminology
Design data and event streaming pipelines that use Connect
Configure and operate Connect environments at scale
Deploy secured and highly available Connect clusters
Build sink and source connectors and single message transforms and converters