kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Krzysztof Ciesielski <krzysztof.ciesiel...@softwaremill.pl>
Subject New Consumer API + Reactive Kafka
Date Wed, 02 Dec 2015 14:44:30 GMT

I’m the main maintainer of Reactive Kafka - a wrapper library that provides Kafka API as
Reactive Streams (https://github.com/softwaremill/reactive-kafka).
I’m a bit concerned about switching to Kafka 0.9 because of the new Consumer API which doesn’t
seem to fit well into this paradigm, comparing to the old one. My main concerns are:

1. Our current code uses the KafkaIterator and reads messages sequentially, then sends them
further upstream. In the new API, you cannot control how many messages are returned with poll(),
so we would need to introduce some kind of in-memory buffering.
2. You cannot specify which offsets to commit. Our current native committer (https://github.com/softwaremill/reactive-kafka/blob/4055e88c09b8e08aefe8dbbd4748605df5779b07/core/src/main/scala/com/softwaremill/react/kafka/commit/native/NativeCommitter.scala)
uses the OffsetCommitRequest/Response API and kafka.api.ConsumerMetadataRequest/Response
for resolving brokers. Switching to Kafka 0.9 brings some compilation errors that raise questions.

My questions are:

1. Do I understand the capabilities and limitations of new API correctly? :)
2. Can we stay with the old iterator-based client, or is it going to get abandoned in future
Kafka versions, or discouraged for some reasons?
3. Can we still use the OffsetCommitRequest/Response API to commit messages manually? If yes,
could someone update this example: https://cwiki.apache.org/confluence/display/KAFKA/Committing+and+fetching+consumer+offsets+in+Kafka or
give me a few hints on how to do this with 0.9?

By the way, we’d like our library to appear on the Ecosystem Wiki, I’m not sure how to
request that officially :)

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message