WORK-IN-PROGRESS
Streaming data changes to a Data Lake with Debezium and Delta Lake pipeline (Medium.com)
This is an example end-to-end project that demonstrates the Debezium-Delta Lake combo pipeline
See medium post for more details
- Debezium reads database logs, produces json messages that describe the changes and streams them to Kafka
- Kafka streams the messages and stores them in a S3 folder. We call it Bronze table as it stores raw messages
- Using Spark with Delta Lake we transform the messages to INSERT, UPDATE and DELETE operations, and run them on the target data lake table. This is the table that holds the latest state of all source databases. We call it Silver table
- Next we can perform further aggregations on the Silver table for analytics. We call it Gold table
- compose: Docker-Compose configuration that deploys containers with Debezium stack (Kafka, Zookeepr and Kafka-Connect), reads changes from the source databases and streams them to S3
- voter-processing: Notebook with PySpark code that transforms Debezium messages to INSERT, UPDATE and DELETE operations
- fake_it: For an end-to-end example, a simulator of a voters book application's database with live input
- analytics: a spark job that simulates reading all history versions from delta lake, and then storing the most updated data, for each poll.
- export DEBEZIUM_VERSION=1.0
- cd compose
- docker-compose up -d
curl -i -X POST -H "Accept:application/json" -H "Content-Type:application/json" http://localhost:8084/connectors/ -d @debezium/config.json
Import the notebook file in \voter-processing\voter-processing.html to a Databricks Community account and follow the instructions inside the notebook
https://community.cloud.databricks.com/
- Change the voter-processing from notebook to PySpark application
- Add the PySpark application to the Docker-Compose
- Change the configurations so that Kafka writes to local file system instead of S3
- Change the Spark application so that it read Kafka's output instead of generating it's own mock data
Make it a configurable generic tool that can be assembled on top of any supported database