kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Bruno D. Rodrigues" <bruno.rodrig...@litux.org>
Subject Re: Consumer doesn't start picking existing messages from topic until new data is added
Date Thu, 17 Oct 2013 13:17:46 GMT
try this, not sure if it would help though

        props.put("auto.offset.reset", "smallest");

A 17/10/2013, às 14:13, Tarang Dawer <tarang.dawer@gmail.com> escreveu:

> Hi All
> 
> I am facing issues with the kafka 0.8 consumer with the consumer not
> picking existing messages in the topic for consumption.
> 
> 
> 
> Following is a snippet from my code  : -
> 
> 
> public void run() {
>            ConsumerIterator<byte[],byte[]> it = stream.iterator();
>            System.out.println("************************In Run : before
> while loop");   //*1st SYSO*
>            while (it.hasNext()) {
>                byte[] message = it.next().message();
>                System.out.println("Got New Mesage , Processing .. Dump in
> hbase table....********************************");   //*2nd SYSO*
> }
> 
> 
> Steps: -
> 1. Cleaned  zookeeper and kafka logs.
> 2. Started both again and produced 1000 messages in the topic.
> 3. Started the consumer.  the consumer , i got log till  *1st SYSO* but no
> progress. it kept on waiting for the stream to get messages
> 4. Produced 1000 more messages in the topic, now the data consumption
> starts , but the messages which i produced earlier in the topic were not
> consumed, as the hbase table showed only 1000 messages , which increased
> further depending on the additional number of messages i produced further,
> but however, the 1st initial deficit is still there.
> 
> As it seems to me, the 1st 1000 messages are being lost, although the topic
> is getting created in the kafka-log directory , with the log file showing
> the initial messages there in the logbefore i start the consumer) , but the
> consumer does not seems to be picking them up.
> 
> I am using Zookeeper-3.4.5 .
> 
> Kafka-Settings are default , except that i the partitions for each topic
> are set to 1.
> 
> Please help me out.
> 
> 
> Thanks
> Tarang Dawer


Mime
View raw message