spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Diwakar Dhanuskodi <diwakar.dhanusk...@gmail.com>
Subject Re: KafkaUtils.createStream not picking smallest offset
Date Sun, 14 Aug 2016 01:52:18 GMT

Not using  check  pointing now.  Source is  producing 1.2million messages to topic. We
 are  using  zookeeper offsets for other  downstreams  too. That's  the  reason  going
 with  createstream which  stores offsets in zookeeper. 


Sent from Samsung Mobile.

<div>-------- Original message --------</div><div>From: Cody Koeninger <cody@koeninger.org>
</div><div>Date:12/08/2016  23:42  (GMT+05:30) </div><div>To: Diwakar
Dhanuskodi <diwakar.dhanuskodi@gmail.com>, user@spark.apache.org </div><div>Cc:
 </div><div>Subject: Re: KafkaUtils.createStream not picking smallest offset </div><div>
</div>Are you checkpointing?


Beyond that, why are you using createStream instead of createDirectStream

On Fri, Aug 12, 2016 at 12:32 PM, Diwakar Dhanuskodi
<diwakar.dhanuskodi@gmail.com> wrote:
> Okay .
> I could  delete  the  consumer group in zookeeper and  start  again to  re
> use same consumer group name. But  this  is  not  working  though . Somehow
> createstream  is  picking  the  offset from  some where other than
> /consumers/ from  zookeeper
>
>
> Sent from Samsung Mobile.
>
>
>
>
>
>
>
>
> -------- Original message --------
> From: Cody Koeninger <cody@koeninger.org>
> Date:12/08/2016 18:02 (GMT+05:30)
> To: Diwakar Dhanuskodi <diwakar.dhanuskodi@gmail.com>
> Cc:
> Subject: Re: KafkaUtils.createStream not picking smallest offset
>
> Auto offset reset only applies if there aren't offsets available otherwise.
>
> The old high level consumer stores offsets in zookeeper.
>
> If you want to make sure you're starting clean, use a new consumer group
> I'd.
>
> On Aug 12, 2016 3:35 AM, "Diwakar Dhanuskodi" <diwakar.dhanuskodi@gmail.com>
> wrote:
>>
>>
>> Hi,
>> We are  using  spark  1.6.1 and  kafka 0.9.
>>
>> KafkaUtils.createStream is  showing strange behaviour. Though
>> auto.offset.reset is  set  to  smallest .  Whenever we  need  to  restart
>> the  stream it  is  picking up  the  latest  offset which  is not  expected.
>> Do  we  need  to  set  any  other  properties ?.
>>
>> createDirectStream works fine  in  this  above  case.
>>
>>
>> Sent from Samsung Mobile.
Mime
View raw message