spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jungtaek Lim <kabh...@gmail.com>
Subject Kafka offset committer tool for structured streaming query
Date Mon, 23 Sep 2019 15:59:32 GMT
Hi Spark users, especially Structured Streaming users who are using Kafka
as data source,

I'm pleased to introduce Kafka offset committer, which enables commit
offsets which batch has been processed. The tool is basically an
implementation of streaming query listener, which listens for events and
commit offsets for each batch. Please refer README.md in the repository to
see more details.

Here's Github repository for this tool:
https://github.com/HeartSaVioR/spark-sql-kafka-offset-committer
Currently it hasn't be published to Maven central, so you might need to
build the source and add jar via "--jars" option until artifact is
published.
I'd be happy to hear new ideas of improvements, and much appreciated for
contributions!

Enjoy!

Thanks,
Jungtaek Lim (HeartSaVioR)

Mime
View raw message