kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mukesh Jha <me.mukesh....@gmail.com>
Subject KafkaUtils not consuming all the data from all partitions
Date Wed, 07 Jan 2015 14:00:09 GMT
Hi Guys,

I have a kafka topic having 90 partitions and I running
SparkStreaming(1.2.0) to read from kafka via KafkaUtils to create 10
kafka-receivers.

My streaming is running fine and there is no delay in processing, just that
some partitions data is never getting picked up. From the kafka console I
can see that each receiver is consuming data from 9 partitions but the lag
for some offsets keeps on increasing.

Below is my kafka-consumers parameters.

Any of you have face this kind of issue, if so then do you have any
pointers to fix it?

Map<String, String> kafkaConf = new HashMap<String, String>();
kafkaConf.put("zookeeper.connect", kafkaZkQuorum);
kafkaConf.put("group.id", kafkaConsumerGroup);
kafkaConf.put("consumer.timeout.ms", "30000");
kafkaConf.put("auto.offset.reset", "largest");
kafkaConf.put("fetch.message.max.bytes", "20000000");
kafkaConf.put("zookeeper.session.timeout.ms", "6000");
kafkaConf.put("zookeeper.connection.timeout.ms", "6000");
kafkaConf.put("zookeeper.sync.time.ms", "2000");
kafkaConf.put("rebalance.backoff.ms", "10000");
kafkaConf.put("rebalance.max.retries", "20");

-- 
Thanks & Regards,

*Mukesh Jha <me.mukesh.jha@gmail.com>*

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message