Project
Structured logger
A metadata-driven structured logging framework: it generates type-safe loggers for Java and Python from JSON definitions and lands every log line in Apache Iceberg tables you can query with SQL.
Logs are data, so this framework treats them that way. You describe a log record once in JSON; the generator produces type-safe loggers for Java and Python; and every record flows into Apache Iceberg tables that can be queried with Trino.
What it does
-
Generates type-safe Java and Python loggers from JSON configurations.
-
Ships records through Kafka (JSON or Avro with a schema registry) or a sidecar to S3 (Avro or Parquet with Zstd), selectable per log type.
-
Ingests into Iceberg tables on S3/MinIO with Spark streaming in 10-second micro-batches — ACID transactions, schema evolution, and time travel included.
-
Queries everything with SQL through Trino.
How it is built
Java and Python generators and runtimes; Kafka; Spark Structured Streaming consumers that detect the encoding; Iceberg on MinIO; Trino. A benchmark harness compares five delivery approaches — Kafka with JSON delivered about 7 ms median latency at about $0.11 per million rows, while a sidecar writing Parquet to S3 was about fifteen times cheaper at about 12 seconds median latency. The repository also includes a self-contained Apache Flink Table API tutorial.
Its sister project is warehouse-sync.