kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "M. Manna" <manme...@gmail.com>
Subject Retrieving a specific message for re-delivery without moving head using seek()
Date Tue, 17 Apr 2018 11:26:20 GMT
Hello,

I am using Kafka in plain producer-consumer style. I have a daily exchange
of about 1 million messages avg 50kb size per message. I am still using
0.10.2.1

Whilst I am relying on Kafka's stability under a 9-node configuration - I
would like to build some internal process which can keep my messages in
case of "Unreachable" broker/host issue. I will be able to resend those
messages provided that I can query them without moving the pointer.

>From my last encounter, I remember that using topic/partition/offset
details fetched using get() future call on new producer API, I can get
details and use them in seek() to retrieve details. However, seek() will
move the head as I understand from javadocs here here.
<https://kafka.apache.org/0102/javadoc/org/apache/kafka/clients/consumer/KafkaConsumer.html>

To explain my problem better, lets look at the following:

1) The producer failed to send some messages - no future received. This is
due to broker outage or a disaster.
2) These pending messages are logged in DB or in some file.
3) The disaster is resolved - all systems are back up.
4) New messages are being generated and sent/received successfully.
5) But my message @ step 1 is still pending.

For me, as long as I can query those messages and resend them it's all
good. Currently, I simply store them in a DB upon failure, retrieve them
when everything's back up, and resend them.

I was wondering if someone can confirm whether v.11 will have something to
ease this process, or should I submit a KIP for this?

Regards,

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message