kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Charlie Knudsen <charlie.knud...@smartthings.com>
Subject Re: Is there a way to know when I've reached the end of a partition (consumed all messages) when using the high-level consumer?
Date Sat, 09 May 2015 17:42:13 GMT
Hi James,
What are you trying to do exactly? If all you are trying to do is monitor
how far behind a consumer is getting you could use the ConsumerOffsetChecker.
As described in the link below.
http://community.spiceworks.com/how_to/77610-how-far-behind-is-your-kafka-consumer

Each message being processed will also have the offset and partition
attached to it so with that data. I suppose that information plus info from
a fetch response you could determine this with in an application.
https://cwiki.apache.org/confluence/display/KAFKA/A+Guide+To+The+Kafka+Protocol#AGuideToTheKafkaProtocol-FetchResponse

Does that help?


On Fri, May 8, 2015 at 6:04 PM, James Cheng <jcheng@tivo.com> wrote:

> Hi,
>
> I want to use the high level consumer to read all partitions for a topic,
> and know when I have reached "the end". I know "the end" might be a little
> vague, since items keep showing up, but I'm trying to get as close as
> possible. I know that more messages might show up later, but I want to know
> when I've received all the items that are currently available in the topic.
>
> Is there a standard/recommended way to do this?
>
> I know one way to do it is to first issue an OffsetRequest for each
> partition, which would get me the last offset, and then use that
> information in my high level consumer to detect when I've reached that a
> message with that offset. Which is exactly what the SimpleConsumer example
> does (
> https://cwiki.apache.org/confluence/display/KAFKA/0.8.0+SimpleConsumer+Example).
> That involves finding the leader for the partition, etc etc. Not hard, but
> a bunch of steps.
>
> I noticed that kafkacat has an option similar to what I'm looking for:
>   -e                 Exit successfully when last message received
>
> Looking at the code, it appears that a FetchRequest returns the
> HighwaterMarkOffset mark for a partition, and the API docs confirm that:
> https://cwiki.apache.org/confluence/display/KAFKA/A+Guide+To+The+Kafka+Protocol#AGuideToTheKafkaProtocol-FetchResponse
>
> Does the Java high-level consumer expose the HighwaterMarkOffset in any
> way? I looked but I couldn't find such a thing.
>
> Thanks,
> -James
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message