kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Tomas Niño Kehoe <tomasninoke...@gmail.com>
Subject Retrieving "lost" messages produced while the consumer was down.
Date Tue, 21 Jul 2015 21:24:17 GMT
Hi,

We've been using Kafka for a couple of months, and now we're trying to to
write a Simple application using the ConsumerGroup to fully understand
Kafka.

Having the producer continually writing data, our consumer occasionally
needs to be restarted. However, once the program is brought back up,
messages which we're produced during that period of time are not being
read. Instead, the consumer (this is a single consumer inside a Consume
group) will read the messages produced after it was brought back up.  Its
configuration doesn't change at all.

For example using the simple consumer/producer apps:

Produced 1, 2, 3, 4, 5
Consumed 1, 2, 3, 4, 5

[Stop the consumer]
Produce 20, 21, 22, 23

When the consumer is brought back up, I'd like to get 20, 21, 22, 23, but I
will only get either new messages, or all the messages using
(--from-beginning).

Is there a way of achieving this programatically, without for example
writing an offset into the zookeeper node? Is the OffsetCommitRequest the
way to go?

Thanks in advance


Tomás

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message