kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From David Black <david.bl...@workday.com>
Subject Kafka log4j appender infinite logging cycle question
Date Thu, 01 Aug 2013 10:41:21 GMT

I have the following scenario:

- using the Kafka log4j appender (in fact its a variant on the OOTB

- configuring the appender for the root log4j category

- appender is using the async producer

I am wondering about what happens when e.g. the producer cannot connect to
any broker and it logs error messages, which (I’m guessing) get passed by
log4j back into the (kafka) appender. I think there is a possibility of a
kind of infinite cycle, where any log message emitted by the kafka producer
framework may result in more messages being emitted by the kafka producer
framework, due to being pushed back through the appender and so on.

To an extent I can control this with log4j.additivity e.g. by having a
separate category for the kafka root package (direct its output to my other
appenders but not the kafka appender), and setting additivity for kafka
root package to false. But I think messages from other packages (from other
libraries) that the kafka producer uses may still cause this cycling. Would
I have to identify all those packages and also exclude them? I think this
would not work because other parts of my process use some of those same
jars and I need their output to go to kafka via the kafka log4j appender.

I’m wondering has anyone else encountered / considered / dealt with this
and if so how?

Any help much appreciated

David Black

Workday Inc.

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message