kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ali Akhtar <ali.rac...@gmail.com>
Subject Tracking when a batch of messages has arrived?
Date Fri, 02 Dec 2016 13:02:54 GMT

I need to send a group of messages, which are all related, and then process
those messages, only when all of them have arrived.

Here is how I'm planning to do this. Is this the right way, and can any
improvements be made to this?

1) Send a message to a topic called batch_start, with a batch id (which
will be a UUID)

2) Post the messages to a topic called batch_msgs_<batch_id>. Here batch_id
will be the batch id sent in batch_start.

The number of messages sent will be recorded by the producer.

3) Send a message to batch_end with the batch id and the number of sent

4) On the consumer side, using Kafka Streaming, I would listen to batch_end.

5) When the message there arrives, I will start another instance of Kafka
Streaming, which will process the messages in batch_msgs_<batch_id>

6) Perhaps to be extra safe, whenever batch_end arrives, I will start a
throwaway consumer which will just count the number of messages in
batch_msgs_<batch_id>. If these don't match the # of messages specified in
the batch_end message, then it will assume that the batch hasn't yet
finished arriving, and it will wait for some time before retrying. Once the
correct # of messages have arrived, THEN it will trigger step 5 above.

Will the above method work, or should I make any changes to it?

Is step 6 necessary?

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message