kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Bhavesh Mistry <mistry.p.bhav...@gmail.com>
Subject Re: QOS on Producer Side
Date Tue, 06 May 2014 18:30:19 GMT
So to retrieve the message that was not transmitted due to error, I will
have to store the message in each call back the message and store this in a
file.    Is there an alternative solution ?

What is best way to achieve QOS on producer side ?

Also,  will new Producer API will work with Broker version  0.8.0 ?  Please
let me know.

Thanks,

Bhavesh


On Tue, May 6, 2014 at 8:46 AM, Jun Rao <junrao@gmail.com> wrote:

> The callback will be invoked on completion. The exception tells you whether
> the send completed or not. This api will be available in the next release.
>
> Thanks,
>
> Jun
>
>
> On Mon, May 5, 2014 at 10:03 PM, Bhavesh Mistry
> <mistry.p.bhavesh@gmail.com>wrote:
>
> > Thanks for answers.
> >
> > Does the callback get call on failure only or for success as well ?
>  Also,
> > how do I do this on Kafka 0.8.0 ?   Is there any plan for  adding
> buffering
> > on disk for next version ?  Also, when application restart, kafka
> producer
> > will have to transmit the messages (buffered messages on disk)  ?
> >
> > I can not upgrade to Kafka 0.8.1.1 right now.  how can I achieve this on
> > Kafka version 0.8.0.  Also, when retransmitting the messages I wanted to
> > transfer dedicated Kafka back-up or reserved partition for topic
> (near-real
> > time topic).
> >
> > How would you handle retransmission of messages ?
> >
> > Any idea about implementation will be very helpful.
> >
> > Thanks,
> >
> > Bhavesh
> >
> > On Mon, May 5, 2014 at 9:29 PM, Jun Rao <junrao@gmail.com> wrote:
> >
> > > At LinkedIn, we only buffer messages in memory in the producer. We try
> as
> > > much as we can to make the Kafka cluster always available (with
> > > replication). The "on failure" feature you mentioned can be implemented
> > by
> > > providing a callback. You can see the example in the javadoc.
> > >
> > > *send
> > > <
> > >
> >
> http://empathybox.com/kafka-javadoc/kafka/clients/producer/KafkaProducer.html#send(kafka.clients.producer.ProducerRecord
> > > ,
> > > kafka.clients.producer.Callback)>*(ProducerRecord<
> > >
> >
> http://empathybox.com/kafka-javadoc/kafka/clients/producer/ProducerRecord.html
> > > >
> > >  record, Callback<
> > >
> http://empathybox.com/kafka-javadoc/kafka/clients/producer/Callback.html
> > >
> > >  callback)
> > >
> > > Thanks,
> > >
> > > Jun
> > >
> > > On Mon, May 5, 2014 at 6:30 PM, Bhavesh Mistry
> > > <mistry.p.bhavesh@gmail.com>wrote:
> > >
> > > > We are using Kafka for operation metrics and we do not want to loose
> > any
> > > > data at all if there is issue with Network or all brokers needs to be
> > > > rebooted or operation reason down time while all producers are still
> > > > producing data on front end side.  We use async to publish messages
> and
> > > we
> > > > are using Kafka version 0.8.0.
> > > >
> > > >
> > > >
> > > > Has any one implemented buffering on local disk (on producer side)
> and
> > > > transmit messages when network connection is restored?   How do I get
> > > > handle to list of messages async thread could not transfer after x
> > reties
> > > > ?  I know new producer API has callback interface, but is per message
> > not
> > > > per producer instance  ?
> > > >
> > > >
> > > >
> > > > Is this final new Producer API ?
> > > >
> > > >
> > >
> >
> http://empathybox.com/kafka-javadoc/index.html?kafka/clients/producer/KafkaProducer.html
> > > >
> > > >
> > > > is there a plan to add method like.  onFailure(List<Messages>
> messages,
> > > > Exception exception ) ?
> > > >
> > > >
> > > >
> > > >
> > > > Basically, I have to address the QOS on producer side, and be able to
> > > > buffer on disk and retransmit all message to partitions that are
> > reserved
> > > > for messages that happened in past.
> > > >
> > > >
> > > >
> > > > How does Linked-in handle QOS on producer side ?
> > > >
> > > >
> > > > Is there any plan to add this QOS feature on Producer Side with
> > > Strategies
> > > > to
> > > > store and retransmit the message ?  If we do get the list of messages
> >  is
> > > > call back, will it be compressed data  ?
> > > >
> > > >
> > > >
> > > > I would appreciate Kafka Developers and others feedback on how to
> > > implement
> > > > QOS.
> > > >
> > > >
> > > > Thanks,
> > > >
> > > >
> > > > Bhavesh
> > > >
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message