kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Aniket Bhatnagar <aniket.bhatna...@gmail.com>
Subject Re: Delayed Queue
Date Sun, 23 Feb 2014 10:14:46 GMT
Alright. Here is an approach that I can think of. Create a topic for each
delay and produce message to the appropriate topic. Each produced message
should have a timestamp at which it was pushed to the queue. At the
consumer side, fetch a message from a partition and compare the message
timestamp with system's timestamp to see if enough time has passed for you
to process the message. If enough time has passed, process the message and
commit the message's offset otherwise make sure you do not commit the
offset.

You may have to use low level consumer API to achieve this, however double
check if you can use high level consumer API to achieve the same (as I am
not an expert of the consumer APIs).

Also,  this assumes the size of your delay set is small as it is
recommended to only have couple of thousand partitions.
 On 23 Feb 2014 15:15, "Jagan Ranganathan" <jagan@zohocorp.com> wrote:

> Hi Aniket,
>
>
> I am looking at "Finite set of delays" and messages will use one on the
> defined set.
>
>
> Regards,
> Jagan
>
> ---- On Sun, 23 Feb 2014 15:13:02 +0530 Aniket
> Bhatnagar&lt;aniket.bhatnagar@gmail.com&gt; wrote ----
>
>
> Hi Jagan
>
> Are you expecting each message to have the same delay? Or is there a finite
> set of delays? Or messages can arbitrarily any delay?
>
> Thanks,
> Aniket
>  On 23 Feb 2014 14:13, "Jagan Ranganathan" &lt;jagan@zohocorp.com&gt;
> wrote:
>
> &gt; Dear All,
> &gt;
> &gt; I am looking for a Delayed Queue implementation for my distributed
> system
> &gt; requirement. Would like to know if I can achieve the same using Kafka.
> &gt;
> &gt;
> &gt; Regards,
> &gt; Jagan
> &gt;
> &gt;
>
>
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message