Sequin: Coordinating Change Data Capture in Postgres with Watermarks
2025-01-05
Sequin is a real-time change data capture (CDC) tool that streams changes from Postgres to destinations like Kafka and SQS. This article dives into how Sequin elegantly solves the complex problem of simultaneously performing full table capture and incremental change capture. By employing a watermarking mechanism, Sequin coordinates two data streams, preventing data loss or duplication and ensuring data consistency. Sequin uses a chunked capture strategy, processing tables in smaller batches for efficiency and reduced memory usage.