kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Madhavi Sreerangam <madhavi.sreeran...@gmail.com>
Subject Issue with log4j Kafka Appender.
Date Wed, 17 Jun 2015 19:44:34 GMT
I have configured my log4j with Kafka Appender.(Kafka version 0.8.2.0)
Following are entries from my log4j file

log4j.appender.KAFKA=kafka.producer.KafkaLog4jAppender
log4j.appender.KAFKA.BrokerList=iakafka301p.dev.ch3.s.com:9092,
iakafka302p.dev.ch3.s.com:9092,iakafka303p.dev.ch3.s.com:9092
log4j.appender.KAFKA.Topic=dev-1.0_audit
log4j.appender.KAFKA.Serializer=kafka.test.AppenderStringSerializer
log4j.appender.KAFKA.layout=org.apache.log4j.PatternLayout
log4j.appender.KAFKA.layout.ConversionPattern=%m-%d

Kafka is configured with 3 servers, 3 partitions and 3 replicas.
I have created a test method to publish the messages to kafka topic as
follows

    private void testKAFKAlog(int noOfMessages){
        for(int i=0; i < noOfMessages; i++){
            KAFKA_LOG.info("Test Message: " + i);
        }
    }
I could not see any messages published into the topic. Then I have modified
the test method to introduce some wait between the requests as follows

    private void testKAFKAlog(int noOfMessages){
        for(int i=0; i < noOfMessages; i++){
            try {
                Thread.sleep(10);
            } catch (InterruptedException e) {
                e.printStackTrace();
            }
            KAFKA_LOG.info("Test Message: " + i);
        }
    }

Then all the messages started publishing. I did this exercise couple of
times with and without sleep between the requests. Messages got published
only when there is sleep in between the requests.
Does any one help me here, what is wrong with the configurations I am
using. (I can't afford 10ms wait for each message, as my application logs
few Million messages for each run).
Is there any way that I can override the default ProducerConfig for log4j
kafka appender.

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message