kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Philip O'Toole <philip.oto...@yahoo.com.INVALID>
Subject Re: [New Feature Request] Ability to Inject Queue Implementation Async Mode
Date Thu, 07 Aug 2014 22:00:30 GMT
Policies for which messages to drop, retain, etc seem like something you should code in your
application. I personally would not like to see this extra complexity added to Kafka. 



> On Aug 7, 2014, at 2:44 PM, Bhavesh Mistry <mistry.p.bhavesh@gmail.com> wrote:
> Basically, requirement is to support message dropping policy in event when
> queue is full.  When you get storm of data (in our case logging due to
> buggy application code), we would like to retain current message instead of
> first one in queue.   We will mitigate this by rate limiting on producer
> side. Only thing is if Kafka allows the flexibility to inject
> implementation, then developer have control over what to drop and retain
> and also what to priorities.
> We would like to change tunable parameters (such as batch size and queue
> size etc or non intrusive parameter that does not impact the life cycle of
> the producer instance )   at runtime after producer instance is created.
> Thanks,
> Bhavesh
>> On Mon, Aug 4, 2014 at 7:05 PM, Joe Stein <joe.stein@stealth.ly> wrote:
>> Is it possible there is another solution to the problem? I think if you
>> could better describe the problem(s) you are facing and how you are
>> architected some then you may get responses from others that perhaps have
>> faced the same problem with similar architectures ... or maybe folks can
>> chime in with solution(s) to the problem(s).  When only being presented
>> with solutions it is hard to say much about if it is problem that folks
>> will have and if this solution will work for them.
>> /*******************************************
>> Joe Stein
>> Founder, Principal Consultant
>> Big Data Open Source Security LLC
>> http://www.stealth.ly
>> Twitter: @allthingshadoop <http://www.twitter.com/allthingshadoop>
>> ********************************************/
>> On Mon, Aug 4, 2014 at 8:52 PM, Bhavesh Mistry <mistry.p.bhavesh@gmail.com
>> wrote:
>>> Kafka Version:  0.8.x
>>> 1) Ability to define which messages get drooped (least recently instead
>> of
>>> most recent in queue)
>>> 2) Try Unbounded Queue to find out the Upper Limit without drooping any
>>> messages for application (use case Stress test)
>>> 3) Priority Blocking Queue ( meaning a single Producer can send messages
>> to
>>> multiple topic and I would like to give Delivery Priority to message for
>>> particular message for topic)
>>> We have use case to support #3 and #1 since we would like to deliver the
>>> Application Heartbeat first then any other event within the queue for any
>>> topics. To lower TCP connections, we only use one producer for 4 topics
>> but
>>> one of the topics has priority for delivery.
>>> Please let me know if this is useful feature to have or not.
>>> Thanks in advance for great support !!
>>> Thanks,
>>> Bhavesh
>>> P.S.  Sorry for asking this question again, but last time there was no
>>> conclusion.

View raw message