kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From mete <efk...@gmail.com>
Subject Re: circuit breaker for producer
Date Mon, 11 May 2015 15:10:03 GMT
Hey Guozhang,

Thanks for the heads up!

Best


On Thu, May 7, 2015 at 1:26 AM, Guozhang Wang <wangguoz@gmail.com> wrote:

> The metrics for checking that would better be "buffer-available-bytes"
> instead of "bufferpool-wait-ratio", checking on its value approaching 0.
>
> Guozhang
>
> On Wed, May 6, 2015 at 3:02 AM, mete <efkarr@gmail.com> wrote:
>
> > Hey Guozhang,
> >
> > I could go with both of the options, eventually i want to detect if there
> > is a problem and isolate it from the rest of the system. And i am trying
> to
> > decide what would be the appropriate metrics to do that.
> >
> > Best
> >
> >
> >
> > On Wed, May 6, 2015 at 6:35 AM, Guozhang Wang <wangguoz@gmail.com>
> wrote:
> >
> > >    1. KAFKA-1955 <https://issues.apache.org/jira/browse/KAFKA-1955>,
I
> > >    think Jay has a WIP patch for it.
> > >    2.
> > >    3.
> > >
> > >
> > > On Tue, May 5, 2015 at 5:10 PM, Jason Rosenberg <jbr@squareup.com>
> > wrote:
> > >
> > > > Guozhang,
> > > >
> > > > Do you have the ticket number for possibly adding in local log file
> > > > failover? Is it actively being worked on?
> > > >
> > > > Thanks,
> > > >
> > > > Jason
> > > >
> > > > On Tue, May 5, 2015 at 6:11 PM, Guozhang Wang <wangguoz@gmail.com>
> > > wrote:
> > > >
> > > > > Does this "log file" acts as a temporary disk buffer when broker
> > slows
> > > > > down, whose data will be re-send to broker later, or do you plan
to
> > use
> > > > it
> > > > > as a separate persistent storage as Kafka brokers?
> > > > >
> > > > > For the former use case, I think there is an open ticket for
> > > integrating
> > > > > this kind of functionality into producer; for the latter use case,
> > you
> > > > may
> > > > > want to do this traffic control out of Kafka producer, i.e. upon
> > > > detecting
> > > > > producer buffer full, do not call send() on it for a while but
> write
> > > to a
> > > > > different file, etc.
> > > > >
> > > > > Guozhang
> > > > >
> > > > > On Tue, May 5, 2015 at 11:28 AM, mete <efkarr@gmail.com> wrote:
> > > > >
> > > > > > Sure, i kind of count on that actually, i guess with this setting
> > the
> > > > > > sender blocks on allocate method and this bufferpool-wait-ratio
> > > > > increases.
> > > > > >
> > > > > > I want to fully compartmentalize the kafka producer from the
rest
> > of
> > > > the
> > > > > > system. Ex: writing to a log file instead of trying to send
to
> > kafka
> > > > when
> > > > > > some metric in the producer indicates that there is a performance
> > > > > > degradation or some other problem.
> > > > > > I was wondering what would be the ideal way of deciding that?
> > > > > >
> > > > > >
> > > > > >
> > > > > > On Tue, May 5, 2015 at 6:32 PM, Jay Kreps <jay.kreps@gmail.com>
> > > wrote:
> > > > > >
> > > > > > > Does block.on.buffer.full=false do what you want?
> > > > > > >
> > > > > > > -Jay
> > > > > > >
> > > > > > > On Tue, May 5, 2015 at 1:59 AM, mete <efkarr@gmail.com>
wrote:
> > > > > > >
> > > > > > > > Hello Folks,
> > > > > > > >
> > > > > > > > I was looking through the kafka.producer metrics on
the JMX
> > > > > interface,
> > > > > > to
> > > > > > > > find a good indicator when to "trip" the circuit.
So far it
> > seems
> > > > > like
> > > > > > > the
> > > > > > > > "bufferpool-wait-ratio" metric is a useful decision
mechanism
> > > when
> > > > to
> > > > > > cut
> > > > > > > > off the production to kafka.
> > > > > > > >
> > > > > > > > As far as i experienced, when kafka server slow for
some
> > reason,
> > > > > > requests
> > > > > > > > start piling up on the producer queue and if you are
not
> > willing
> > > to
> > > > > > drop
> > > > > > > > any messages on the producer, send method starts blocking
> > because
> > > > of
> > > > > > the
> > > > > > > > slow responsiveness.
> > > > > > > >
> > > > > > > > So this buffer pool wait ratio starts going up from
0.x up to
> > > 1.0.
> > > > > And
> > > > > > i
> > > > > > > am
> > > > > > > > thinking about tripping the circuit breaker using
this
> metric,
> > > ex:
> > > > if
> > > > > > > > wait-ratio > 0.90 etc...
> > > > > > > >
> > > > > > > > What do you think? Do you think there would be a better
> > indicator
> > > > to
> > > > > > > check
> > > > > > > > the health overall?
> > > > > > > >
> > > > > > > > Best
> > > > > > > > Mete
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > -- Guozhang
> > > > >
> > > >
> > >
> > >
> > >
> > > --
> > > -- Guozhang
> > >
> >
>
>
>
> --
> -- Guozhang
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message