spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Julia Wistance <>
Subject Kafka Consumer Pre Fetch Messages + Async commits
Date Sat, 26 Aug 2017 06:07:31 GMT
Hi Experts,

A question on what could potentially happen with Spark Streaming 2.2.0 +
Kafka. LocationStrategies says that "new Kafka consumer API will pre-fetch
messages into buffers.".
If we store offsets in Kafka, currently we can only use a async commits.

1 - Could it happen that we commit offsets that we havent processed yet but
the kafka consumers has prefetched
2 - Are there plans to support a sync commit? Although we can go for an
alternate store of commits like HBase / Zookeeper, MySQL etc the code would
wait till the offsets are stored in either of these systems. It would make
sense that Spark / Kafka also adds a sync commit option?

Appreciate the reply.

View raw message