storm-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jason Kania <jason.ka...@ymail.com>
Subject Re: Question About Emitted, Transferred and Acked Bolts
Date Mon, 23 Feb 2015 21:16:35 GMT
Michael,
That's good to know. I was unaware. That said, if execution of a bolt has not occurred, I
would still expect a 0 emit count and acks not to be falling behind the emits by much. My
acks are half my emits.
      From: Michael Rose <michael@fullcontact.com>
 To: "user@storm.apache.org" <user@storm.apache.org>; Jason Kania <jason.kania@ymail.com>

 Sent: Monday, February 23, 2015 3:52 PM
 Subject: Re: Question About Emitted, Transferred and Acked Bolts
   
Keep in mind that those metrics are sampled at the rate of topology.stats.sample.rate, 0.05
by default. If you turn it up to 1.0 you'll see full-resolution, though at the price of more
time spent collecting metrics.
Michael RoseSenior Software EngineerFullContact | fullcontact.comm: +1.720.837.1357 | t: @xorlev

All Your Contacts, Updated and In One Place.Try FullContact for Free


On Mon, Feb 23, 2015 at 12:14 PM, Jason Kania <jason.kania@ymail.com> wrote:

I have two comments to add:
1) Is there any JIRA for invalid metrics values? I did not see one. I am running with bolts
having breakpoints and long before my bolts are every entered, the metrics indicate that these
bolts already have more than 100 emits. I have thought to raise a JIRA on this but I am not
sure what I would add for details. Would some specific debug output aid in resolving this?

2) For acks, is there any possibility of adding tracking for acks that happen after a timeout?
I can step into my bolt each time it is called and confirm that it is acking each request,
yet the acks do not match the emits (which should have a 1 to 1 ratio). I am guessing that
this is because the ack happened too late or it might be incorrect metrics total.

I use the STORM UI for processing tracking.

Thanks,
Jason
      From: Nathan Leung <ncleung@gmail.com>
 To: user <user@storm.apache.org> 
 Sent: Monday, February 23, 2015 11:56 AM
 Subject: Re: Question About Emitted, Transferred and Acked Bolts
   
executed = # of times you called executedacked = # of executed tuples that you acked; ideally
this will match executedemitted = # of tuples that you emitted; if you call emit more than
once per execute call this can be higher than execute counttransferred = # of tuples transferred
downstream; if you have 2 bolts subscribing to your bolt, then this count can be higher than
emitted.


On Mon, Feb 23, 2015 at 11:35 AM, Rahul Reddy <Rahul.Reddy@match.com> wrote:

Hi,

Can you guys help me understand difference between emitted, transferred and acked tuples.

In my case every tuple emitted by ablog-filter-bolt will be processed by ablog-flatten-xml-bolt
which will then be written by ablog-hdfs-bolt to hdfs. Ideally all metrics for executed/acked
should match after tuples are emitted from ablog-filter-bolt . I'm not sure why there is so
much discrepancy in emitted/transferredacked tuple count between these bolts although it dosent
show any failed tuples.

Any ideas what I can check and how to interpret metrics correctly?

Thanks
Rahul





   



  
Mime
View raw message