Chris Collins

Project

warehouse-sync

Metadata-driven incremental replication from PostgreSQL into Apache Iceberg — the database counterpart to my structured-logger project.

Started 26 August 2026 · repository

Data and event platforms

warehouse-sync copies operational database tables into an Apache Iceberg warehouse, incrementally and driven entirely by configuration. It is the sister project to structured-logger: same lakehouse stack and config-driven approach, but reading from database tables instead of application log calls.

What it does

  • Reads PostgreSQL tables incrementally using a declared watermark column (WHERE col > last_watermark ORDER BY col LIMIT batch), advancing after each batch.

  • Three write modes per table: append-only, merge (Iceberg MERGE INTO on the primary key), and atomic full replacement.

  • Runs one-shot or as a daemon, with dry-run, inspect, and backfill commands and JSON-Schema-validated configs.

  • Keeps its state (watermarks and run history) in Iceberg tables, queryable with Trino.

How it is built

Java with a pluggable source interface discovered through ServiceLoader — adding a source is one class and one line; MySQL and Debezium stubs prove the seam. A writer factory chooses the write mode, so the engine itself does not care which is running.

Status

Version 0.1.0: PostgreSQL source, all three write modes, both runtimes.

Updates

No updates yet.