storm-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Michael Rose <mich...@fullcontact.com>
Subject Re: Question About Emitted, Transferred and Acked Bolts
Date Mon, 23 Feb 2015 21:24:51 GMT
Do you have enough ackers to keep up with your traffic? How about failures?

*Michael Rose*
Senior Software Engineer
*Full*Contact | fullcontact.com
<https://www.fullcontact.com/?utm_source=FullContact%20-%20Email%20Signatures&utm_medium=email&utm_content=Signature%20Link&utm_campaign=FullContact%20-%20Email%20Signatures>
m: +1.720.837.1357 | t: @xorlev


All Your Contacts, Updated and In One Place.
Try FullContact for Free
<https://www.fullcontact.com/?utm_source=FullContact%20-%20Email%20Signatures&utm_medium=email&utm_content=Signature%20Link&utm_campaign=FullContact%20-%20Email%20Signatures>

On Mon, Feb 23, 2015 at 2:16 PM, Jason Kania <jason.kania@ymail.com> wrote:

> Michael,
>
> That's good to know. I was unaware. That said, if execution of a bolt has
> not occurred, I would still expect a 0 emit count and acks not to be
> falling behind the emits by much. My acks are half my emits.
>
>   ------------------------------
>  *From:* Michael Rose <michael@fullcontact.com>
> *To:* "user@storm.apache.org" <user@storm.apache.org>; Jason Kania <
> jason.kania@ymail.com>
> *Sent:* Monday, February 23, 2015 3:52 PM
>
> *Subject:* Re: Question About Emitted, Transferred and Acked Bolts
>
> Keep in mind that those metrics are sampled at the rate
> of topology.stats.sample.rate, 0.05 by default. If you turn it up to 1.0
> you'll see full-resolution, though at the price of more time spent
> collecting metrics.
>
> *Michael Rose*
> Senior Software Engineer
> *Full*Contact | fullcontact.com
> <https://www.fullcontact.com/?utm_source=FullContact%20-%20Email%20Signatures&utm_medium=email&utm_content=Signature%20Link&utm_campaign=FullContact%20-%20Email%20Signatures>
> m: +1.720.837.1357 | t: @xorlev
>
>
> All Your Contacts, Updated and In One Place.
> Try FullContact for Free
> <https://www.fullcontact.com/?utm_source=FullContact%20-%20Email%20Signatures&utm_medium=email&utm_content=Signature%20Link&utm_campaign=FullContact%20-%20Email%20Signatures>
>
>
>
> On Mon, Feb 23, 2015 at 12:14 PM, Jason Kania <jason.kania@ymail.com>
> wrote:
>
> I have two comments to add:
>
> 1) Is there any JIRA for invalid metrics values? I did not see one. I am
> running with bolts having breakpoints and long before my bolts are every
> entered, the metrics indicate that these bolts already have more than 100
> emits. I have thought to raise a JIRA on this but I am not sure what I
> would add for details. Would some specific debug output aid in resolving
> this?
>
> 2) For acks, is there any possibility of adding tracking for acks that
> happen after a timeout? I can step into my bolt each time it is called and
> confirm that it is acking each request, yet the acks do not match the emits
> (which should have a 1 to 1 ratio). I am guessing that this is because the
> ack happened too late or it might be incorrect metrics total.
>
> I use the STORM UI for processing tracking.
>
> Thanks,
>
> Jason
>
>   ------------------------------
>  *From:* Nathan Leung <ncleung@gmail.com>
> *To:* user <user@storm.apache.org>
> *Sent:* Monday, February 23, 2015 11:56 AM
> *Subject:* Re: Question About Emitted, Transferred and Acked Bolts
>
> executed = # of times you called executed
> acked = # of executed tuples that you acked; ideally this will match
> executed
> emitted = # of tuples that you emitted; if you call emit more than once
> per execute call this can be higher than execute count
> transferred = # of tuples transferred downstream; if you have 2 bolts
> subscribing to your bolt, then this count can be higher than emitted.
>
>
>
> On Mon, Feb 23, 2015 at 11:35 AM, Rahul Reddy <Rahul.Reddy@match.com>
> wrote:
>
> Hi,
>
> Can you guys help me understand difference between emitted, transferred
> and acked tuples.
>
> In my case every tuple emitted by ablog-filter-bolt will be processed by
> ablog-flatten-xml-bolt which will then be written by ablog-hdfs-bolt to
> hdfs. Ideally all metrics for executed/acked should match after tuples are
> emitted from ablog-filter-bolt . I'm not sure why there is so much
> discrepancy in emitted/transferredacked tuple count between these bolts
> although it dosent show any failed tuples.
>
> Any ideas what I can check and how to interpret metrics correctly?
>
> Thanks
> Rahul
>
>
>
>
>
>
>
>

Mime
View raw message