kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jun Rao <jun...@gmail.com>
Subject Re: Producer.send questions
Date Sat, 24 Aug 2013 15:22:06 GMT
You don't need to restart the producer. The producer currently handles all
error/exceptions by refreshing the metadata and retrying. If it fails all
retries, it throws a FailedToSendMessageException to the caller (in sync
mode). The original cause is not included in this exception. We have
thought about being a bit smarter in the producer retry logic such that it
only retries on recoverable errors and could implement this at some point.
Other than FailedToSendMessageException, the producer can also throw
QueueFullException.
This is an indication that the producer is sending data at a rate faster
than the broker can handle. This may or may not be recoverable since it
depends on the load.

Thanks,

Jun


On Sat, Aug 24, 2013 at 1:44 AM, Jason Rosenberg <jbr@squareup.com> wrote:

> Jun,
>
> There are several others I've seen that I would have thought would be
> retryable (possibly after an exponential backoff delay).  I'm curious
> about:
>
> BrokerNotAvailableException
> FailedToSendMessageException
> QueueFullException (happens if producerType is 'async')
> KafkaException (this seems to wrap lots of base conditions, does one have
> to sort through the different wrapped exception types?)
> LeaderNotAvailableException
> MessageSizeTooLargeException (does a batch of messages get treated as a
> single message, when checking for message size too large?)
> ReplicaNotAvailableException
> UnavailableProducerException
> UnknownException
>
> Also, what about my first question, regarding whether it makes sense to
> refresh a producer by closing it and restarting it after a failure?
>
> Thanks,
>
> Jason
>
>
> On Fri, Aug 23, 2013 at 9:07 PM, Jun Rao <junrao@gmail.com> wrote:
>
> > For the most part, only SocketExceptions and
> NotLeaderForPartitionException
> > are recoverable. MessageSizeTooLargeException may be recoverable with a
> > smaller batch size.
> >
> > Thanks,
> >
> > Jun
> >
> >
> > On Fri, Aug 23, 2013 at 4:09 PM, Jason Rosenberg <jbr@squareup.com>
> wrote:
> >
> > > I'm using the kafka.javaapi.producer.Producer class from a java client.
> > >  I'm wondering if it ever makes sense to refresh a producer by stopping
> > it
> > > and creating a new one, for example in response to a downstream IO
> error
> > > (e.g. a broker got restarted, or a stale socket, etc.).
> > >
> > > Or should it always be safe to rely on the producer's implementation to
> > > manage it's pool of BlockingChannel connections, etc.
> > >
> > > I'm also interested in trying to understand which exceptions indicate a
> > > failed send() request might be retryable (basically anything that
> doesn't
> > > involve a data-dependent problem, like a malformed message, or a
> message
> > > too large, etc.).
> > >
> > > Unfortunately, the range of Exceptions that can be thrown by the
> various
> > > javaapi methods is not yet well documented.  It would be nice to have
> > some
> > > notion of whether an exception is the result of a data error, or a
> > > transient downstream connection error, etc.
> > >
> > > Jason
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message