kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Andrey Dyachkov <andrey.dyach...@gmail.com>
Subject Re: Kafka client circuit breaker
Date Mon, 31 Oct 2016 08:17:15 GMT
R Krishna,
A particular set of exceptions has to be defined.
At least what I know: timeout exception, network exception.
I am not sure I got the point about auto-recovery.
About incoming messages, we can return the exception that
the client can not access topic/broker/partition and user can properly
process it.

Dave,
There is a question what the thing is we check failures for. If this is a
topic we can not just block the whole topic because
there could partitions on other brokers which are at least available for
reads. From my point of view checking
per broker is a good enough solution but as I already said it wasteful to
implement it on the user side because
Kafka client does the job to find out the brokers before the message is
gonna be read/written.

Thank you for support!

On Mon, 31 Oct 2016 at 02:41 Tauzell, Dave <Dave.Tauzell@surescripts.com>
wrote:

> Once enough failures happen the circuit is marked open.  The client would
> then periodically try some messages until it works again.  Others would be
> failed.
>
> There are a number of existing circuit breaker libraries you can use in
> the meantime like the Netflix one.
>
> Dave
>
> > On Oct 30, 2016, at 20:23, R Krishna <krishna81m@gmail.com> wrote:
> >
> > So, basically, you want the clients to gracefully handle timeouts and
> short
> > circuit if it is unavailable broker exception and also clients should be
> > able automatically recover when it is up? What about the messages that
> come
> > in when broker is not available?
> >
> > On Sun, Oct 30, 2016 at 12:25 PM, Andrey Dyachkov <
> andrey.dyachkov@gmail.com
> >> wrote:
> >
> >> R Krishna,
> >>
> >> We have default timeout for producer and consumer it is 30 secs.
> >> If Kafka is not available we will wait 30 secs on each request.
> >> Decreasing timeout won't help much because we have a lot requests
> >> but circuiting them will help us tremendous. We could implement such
> >> logic before actually calling Kafka client but there is problem what the
> >> trigger is to open the circuit.
> >> I thought about circuiting by unavailable broker
> >> but it is quite an expansive operation to figure out in the client's
> code
> >> on the flip side Kaka client already does all the job to make out the
> >> broker.
> >>
> >> For us very important to stay live even if Kafka is unavailable (it
> >> happens) that's why I think avoiding cascading issues will be very
> useful.
> >>
> >>> On Sat, 29 Oct 2016 at 19:22 R Krishna <krishna81m@gmail.com> wrote:
> >>>
> >>> What are the issues and problems you want to solve though?
> >>> On Oct 29, 2016 3:59 AM, "Andrey Dyachkov" <andrey.dyachkov@gmail.com>
> >>> wrote:
> >>>
> >>>> Hi,
> >>>>
> >>>> We use Kafka for around half a year as a main backend system for
> >> storing
> >>>> events from big variety of internal microservices. In general It works
> >>>> quite well and we experience issues only in some rare 'corner cases'
> >>> where
> >>>> we either run against a bug or do not understand why Kafka behaves in
> >>>> particular manner. The fact Kafka is the main system is used under our
> >>>> software If it fails our system will fail too. That's why I am writing
> >> to
> >>>> ask about the solution which can help us tolerate different issues on
> >>> Kafka
> >>>> side. First of all could you tell me why doesn't Kafka client have the
> >>>> solution for this kind of problems? Have you ever had thoughts about
> >> it?
> >>>> Thank you in advance!
> >>>> --
> >>>>
> >>>> With great enthusiasm,
> >>>> Andrey
> >>>>
> >>>
> >> --
> >>
> >> With great enthusiasm,
> >> Andrey
> >>
> >
> >
> >
> > --
> > Radha Krishna, Proddaturi
> > 253-234-5657
> This e-mail and any files transmitted with it are confidential, may
> contain sensitive information, and are intended solely for the use of the
> individual or entity to whom they are addressed. If you have received this
> e-mail in error, please notify the sender by reply e-mail immediately and
> destroy all copies of the e-mail and any attachments.
>
-- 

With great enthusiasm,
Andrey

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message