From user-return-6260-apmail-storm-user-archive=storm.apache.org@storm.apache.org Mon Feb 23 20:53:34 2015 Return-Path: X-Original-To: apmail-storm-user-archive@minotaur.apache.org Delivered-To: apmail-storm-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 6A9221782A for ; Mon, 23 Feb 2015 20:53:34 +0000 (UTC) Received: (qmail 71557 invoked by uid 500); 23 Feb 2015 20:53:33 -0000 Delivered-To: apmail-storm-user-archive@storm.apache.org Received: (qmail 71512 invoked by uid 500); 23 Feb 2015 20:53:33 -0000 Mailing-List: contact user-help@storm.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@storm.apache.org Delivered-To: mailing list user@storm.apache.org Received: (qmail 71501 invoked by uid 99); 23 Feb 2015 20:53:33 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 23 Feb 2015 20:53:33 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of michael@fullcontact.com designates 209.85.223.179 as permitted sender) Received: from [209.85.223.179] (HELO mail-ie0-f179.google.com) (209.85.223.179) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 23 Feb 2015 20:53:29 +0000 Received: by iecvy18 with SMTP id vy18so26615238iec.13 for ; Mon, 23 Feb 2015 12:52:23 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:date :message-id:subject:from:to:content-type; bh=5pCMUNFRHl1NV+/ljGAsVWvvZTwcX5tgP9lYDugttiY=; b=j321CgCgdht79tRenNASR54RCmDJOC08/fp7nQyWWQgn5FoRxP1DmVL9J93oOlDqzR JRiifT3gN0MVnC2dszLFuRLQfF+bwf2ZAPBAmYGp7VeYcv0Wwev9yW5DfOlxP2uEluem eVqrcEVmKpBcxqz3PlCjwzEbZlmKSKhb7kENE0FmjcB0iXGHJBYYJWaBZ/FonBkZCd8r JEQ3ZJ454FCWV4GQvI7gMg9Rcbly3mqgTua0e3uv1vkrONtMkO6GqIPSRHbBynecLrjV T0u4/cW1g6HayY7jEEfhErinfZMUlFiYT1ROBqkw8JHDOmdNQc+uxvrVxrZ0A/dx+cYL +x7A== X-Gm-Message-State: ALoCoQlsZORg8kmKrSGfw1EQuqyWb5s4WhnyNySCQjTuvPEpAZ284y/TRQPxaqAm1bYfqQ8I02lG MIME-Version: 1.0 X-Received: by 10.50.129.9 with SMTP id ns9mr15414527igb.24.1424724743091; Mon, 23 Feb 2015 12:52:23 -0800 (PST) Received: by 10.64.32.35 with HTTP; Mon, 23 Feb 2015 12:52:23 -0800 (PST) In-Reply-To: <1515467176.8244275.1424718866915.JavaMail.yahoo@mail.yahoo.com> References: <1515467176.8244275.1424718866915.JavaMail.yahoo@mail.yahoo.com> Date: Mon, 23 Feb 2015 13:52:23 -0700 Message-ID: Subject: Re: Question About Emitted, Transferred and Acked Bolts From: Michael Rose To: "user@storm.apache.org" , Jason Kania Content-Type: multipart/alternative; boundary=047d7b343c9c370f53050fc794ff X-Virus-Checked: Checked by ClamAV on apache.org --047d7b343c9c370f53050fc794ff Content-Type: text/plain; charset=UTF-8 Keep in mind that those metrics are sampled at the rate of topology.stats.sample.rate, 0.05 by default. If you turn it up to 1.0 you'll see full-resolution, though at the price of more time spent collecting metrics. *Michael Rose* Senior Software Engineer *Full*Contact | fullcontact.com m: +1.720.837.1357 | t: @xorlev All Your Contacts, Updated and In One Place. Try FullContact for Free On Mon, Feb 23, 2015 at 12:14 PM, Jason Kania wrote: > I have two comments to add: > > 1) Is there any JIRA for invalid metrics values? I did not see one. I am > running with bolts having breakpoints and long before my bolts are every > entered, the metrics indicate that these bolts already have more than 100 > emits. I have thought to raise a JIRA on this but I am not sure what I > would add for details. Would some specific debug output aid in resolving > this? > > 2) For acks, is there any possibility of adding tracking for acks that > happen after a timeout? I can step into my bolt each time it is called and > confirm that it is acking each request, yet the acks do not match the emits > (which should have a 1 to 1 ratio). I am guessing that this is because the > ack happened too late or it might be incorrect metrics total. > > I use the STORM UI for processing tracking. > > Thanks, > > Jason > > ------------------------------ > *From:* Nathan Leung > *To:* user > *Sent:* Monday, February 23, 2015 11:56 AM > *Subject:* Re: Question About Emitted, Transferred and Acked Bolts > > executed = # of times you called executed > acked = # of executed tuples that you acked; ideally this will match > executed > emitted = # of tuples that you emitted; if you call emit more than once > per execute call this can be higher than execute count > transferred = # of tuples transferred downstream; if you have 2 bolts > subscribing to your bolt, then this count can be higher than emitted. > > > > On Mon, Feb 23, 2015 at 11:35 AM, Rahul Reddy > wrote: > > Hi, > > Can you guys help me understand difference between emitted, transferred > and acked tuples. > > In my case every tuple emitted by ablog-filter-bolt will be processed by > ablog-flatten-xml-bolt which will then be written by ablog-hdfs-bolt to > hdfs. Ideally all metrics for executed/acked should match after tuples are > emitted from ablog-filter-bolt . I'm not sure why there is so much > discrepancy in emitted/transferredacked tuple count between these bolts > although it dosent show any failed tuples. > > Any ideas what I can check and how to interpret metrics correctly? > > Thanks > Rahul > > > > > --047d7b343c9c370f53050fc794ff Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
Keep in mind that those metrics are sampled at the rate of= =C2=A0topology.stats.sample.rate, 0.05 by default. If you turn it up to 1.0= you'll see full-resolution, though at the price of more time spent col= lecting metrics.

Micha= el Rose
Senior Software Engineer
FullContact=C2=A0|=C2=A0fullcontact.com
m: +1.720.837.1357=C2=A0| t:=C2=A0@xorlev<= /div>

All Your Contacts, Updated = and In One Place.

On Mon, Feb 23, 2015 at 12:14 PM, Jason Kani= a <jason.kania@ymail.com> wrote:
I have two comments to add:
<= div dir=3D"ltr">
1) Is there a= ny JIRA for invalid metrics values? I did not see one. I am running with bo= lts having breakpoints and long before my bolts are every entered, the metr= ics indicate that these bolts already have more than 100 emits. I have thou= ght to raise a JIRA on this but I am not sure what I would add for details.= Would some specific debug output aid in resolving this?

2) For acks, i= s there any possibility of adding tracking for acks that happen after a tim= eout? I can step into my bolt each time it is called and confirm that it is= acking each request, yet the acks do not match the emits (which should hav= e a 1 to 1 ratio). I am guessing that this is because the ack happened too = late or it might be incorrect metrics total.

I use the STORM UI for processing tracking.

Thanks,

Jason

= From: Nathan Leung <n= cleung@gmail.com>
To:= user <us= er@storm.apache.org>
Sent:<= /span> Monday, February 23, 2015 11:56 AM
Subject: Re: Question About Emitted, Transferred and = Acked Bolts

executed =3D # of times you called executed
acked =3D # = of executed tuples that you acked; ideally this will match executed
emitted =3D # of tuples that you emitted; if you call emit more than onc= e per execute call this can be higher than execute count
transfer= red =3D # of tuples transferred downstream; if you have 2 bolts subscribing= to your bolt, then this count can be higher than emitted.
=


On Mon, Feb 23, 2015 at 11:= 35 AM, Rahul Reddy <Rahul.Reddy@match.= com> wrote:
Hi,

Can you guys help me understand difference between emitted, transferred and= acked tuples.

In my case every tuple emitted by ablog-filter-bolt will be processed by ab= log-flatten-xml-bolt which will then be written by ablog-hdfs-bolt to hdfs.= Ideally all metrics for executed/acked should match after tuples are emitt= ed from ablog-filter-bolt . I'm not sure why there is so much discrepan= cy in emitted/transferredacked tuple count between these bolts although it = dosent show any failed tuples.

Any ideas what I can check and how to interpret metrics correctly?

Thanks
Rahul




<= br> --047d7b343c9c370f53050fc794ff--