kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jun Rao <jun...@gmail.com>
Subject Re: Kafka log4j appender infinite logging cycle question
Date Thu, 01 Aug 2013 15:03:38 GMT
Yes, there is this self loop, but it's not always a 1-to-1 mapping. Since
you use async producers, each N messages corresponds to a Kafka produce
request. So, if you encounter an error while processing the request, you
add one more message (for every N messages). What you have to avoid is per
message level error (e.g., serialization) since that will add one more
message for each incoming message.



On Thu, Aug 1, 2013 at 3:41 AM, David Black <david.black@workday.com> wrote:

> Hi,
> I have the following scenario:
> - using the Kafka log4j appender (in fact its a variant on the OOTB
> appender)
> - configuring the appender for the root log4j category
> - appender is using the async producer
> I am wondering about what happens when e.g. the producer cannot connect to
> any broker and it logs error messages, which (I’m guessing) get passed by
> log4j back into the (kafka) appender. I think there is a possibility of a
> kind of infinite cycle, where any log message emitted by the kafka producer
> framework may result in more messages being emitted by the kafka producer
> framework, due to being pushed back through the appender and so on.
> To an extent I can control this with log4j.additivity e.g. by having a
> separate category for the kafka root package (direct its output to my other
> appenders but not the kafka appender), and setting additivity for kafka
> root package to false. But I think messages from other packages (from other
> libraries) that the kafka producer uses may still cause this cycling. Would
> I have to identify all those packages and also exclude them? I think this
> would not work because other parts of my process use some of those same
> jars and I need their output to go to kafka via the kafka log4j appender.
> I’m wondering has anyone else encountered / considered / dealt with this
> and if so how?
> Any help much appreciated
> David Black
> Workday Inc.

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message