kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mukesh Jha <me.mukesh....@gmail.com>
Subject KafkaUtils not consuming all the data from all partitions
Date Wed, 07 Jan 2015 14:00:09 GMT
Hi Guys,

I have a kafka topic having 90 partitions and I running
SparkStreaming(1.2.0) to read from kafka via KafkaUtils to create 10

My streaming is running fine and there is no delay in processing, just that
some partitions data is never getting picked up. From the kafka console I
can see that each receiver is consuming data from 9 partitions but the lag
for some offsets keeps on increasing.

Below is my kafka-consumers parameters.

Any of you have face this kind of issue, if so then do you have any
pointers to fix it?

Map<String, String> kafkaConf = new HashMap<String, String>();
kafkaConf.put("zookeeper.connect", kafkaZkQuorum);
kafkaConf.put("group.id", kafkaConsumerGroup);
kafkaConf.put("consumer.timeout.ms", "30000");
kafkaConf.put("auto.offset.reset", "largest");
kafkaConf.put("fetch.message.max.bytes", "20000000");
kafkaConf.put("zookeeper.session.timeout.ms", "6000");
kafkaConf.put("zookeeper.connection.timeout.ms", "6000");
kafkaConf.put("zookeeper.sync.time.ms", "2000");
kafkaConf.put("rebalance.backoff.ms", "10000");
kafkaConf.put("rebalance.max.retries", "20");

Thanks & Regards,

*Mukesh Jha <me.mukesh.jha@gmail.com>*

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message