storm-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From 이승진 <sweetest...@navercorp.com>
Subject Re: is there a way for kafka spout to read from written zookeeper offset
Date Wed, 03 Dec 2014 04:25:59 GMT
thanks for clear answer, it really helped
 
-----Original Message-----
From: "Harsha"&lt;storm@harsha.io&gt; 
To: &lt;user@storm.apache.org&gt;; 
Cc: 
Sent: 2014-12-02 (화) 22:51:29
Subject: Re: is there a way for kafka spout to read from written zookeeper offset







 
 
if you set spoutConfig.forceFromStart=true it will cause KafkaSpout to ignore the existing
offsets stored in the zookeeper and starts reading from the beginning of the queue.  It makes
sense to have it in for the first time you deploy the topology and for the next runs ( kill
and redeploy topology) make sure you turn off the forceFromStart. 
 
On Mon, Dec 1, 2014, at 09:08 PM, 이승진 wrote: 
thanks for comment. 
 
even though I restart topology and obviously there exists offset information in zookeeper,
"No partition information found, using configuration to determine offset" happens.  
 
I'm using spoutConfig.forceFromStart = true; is this option has something to do with that?
what exactly does that option mean? 
 
 
 
-----Original Message-----
From: "Harsha"&lt;storm@harsha.io&gt; 
To: &lt;user@storm.apache.org&gt;; 
Cc: 
Sent: 2014-12-02 (화) 02:27:58
Subject: Re: is there a way for kafka spout to read from written zookeeper offset 
KafkaSpout will read  last committed offset from zookeeper. If there is no offset in the zookeeper
than only it will use configured startOffsetTime.  Can you check in your worker logs after
restarting your topology for this message "No partition information found, using configuration
to determine offset"   
-Harsha 
 
 
On Mon, Dec 1, 2014, at 01:48 AM, 이승진 wrote:  
Hi all,  
 
Kafkaspout periodically write each partition offset to zookeeper.  
 
and spoutConfig.startOffsetTime=-2 means from the beginning, -1 from the latest offset.  
 
Is there a option to read from last committed consumer offset?  
 
When I restart topology due to bugfix or whatsoever, some of the messages are not processed
by storm because I set it to -1.  
 
Sincerely  


 


 




Mime
View raw message