kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Martin Kleppmann <mar...@kleppmann.com>
Subject [ANN] Bottled Water: PostgreSQL to Kafka replication
Date Thu, 23 Apr 2015 15:28:04 GMT
Hi Kafka users,

I'd like to announce a new open source project, called "Bottled Water", for getting data from
PostgreSQL into Kafka:
http://blog.confluent.io/2015/04/23/bottled-water-real-time-integration-of-postgresql-and-kafka/
https://github.com/confluentinc/bottledwater-pg/

Bottled Water combines a consistent snapshot with logical replication, so you can get a full
dump of your database in Kafka, plus low-latency messages whenever a row is inserted, updated
or deleted in the database. This allows you to write Kafka consumers which maintain a copy
of a database in a downstream system, such as a full-text search index, caches, data warehouse,
HDFS, etc.

Bottled Water uses the new logical decoding feature in Postgres 9.4, and the log compaction
feature in Kafka. Each table in Postgres becomes a topic in Kafka, and the table schema is
automatically converted into an Avro schema for export.

It's an alpha release that is not yet fit for production use, but it's ready for experimentation.
Feedback and contributions welcome!

Martin


Mime
View raw message