storm-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Spico Florin <spicoflo...@gmail.com>
Subject Re: Why tuples fail in spout
Date Fri, 06 May 2016 17:54:53 GMT
hi!
  can you please write the hint parallesim that you are using for each sout
and bolts.
there is  formula on what amount on data you can have unacked nd the amound
of data that you can process in your bolts.
  there is a good book Storm applied where this formul is explained nd
asloong with other goodies.
you can read the book from creating a an account.
https://www.safaribooksonline.com
i cite from the book
so if you have 2 spouts for example with 2 task for each of them and the
max spout pending is 10 the number of possible unacked tuples is:
 2 spouts * 2 tasks * 10 max spout pending = 40 unacked tuples possible

if the number of the possible uncked tuples is lower than total parallelism
you have set for your topology then it coud be a bottleneck.
 the total number of parralellism is the sum of all the maxtasks that you
have set up for your bolts or in case that you have used the
hintparallelims is the sum of all hintparralelsim.

suppose that you have bolt b1 with 10 hint p, and bolt b2 with 12 hp then
the total p is 10hpb1+12hpb2=22thp. so considering the above max speding
setup we are fine becuse we have 40 greater than 22 so max pending is not
an issue.
i really recommend all the book Storm applied. is rhe best book that
explains also the storm internals

i hope it helps.
regards,
florin


On Friday, May 6, 2016, Sai Dilip Reddy Kiralam <
dkiralam@aadhya-analytics.com> wrote:

> Hi All,
>
>
> after increment of parallelism in bolts also spout is failing ? how can I
> know how much time my bolts are taking to complete the tuple processing.I'm
> using kafka spout with 10 partitions so I kept my
> topology.max.spout.pending as 10 .do I need to increase the value more than
> that?if yes to extent i can increase the  number ? as a stupid play i kept
> the value 100 to 1 partition kafka it give me metrics error.
>
>
>
> *Best regards,*
>
> *K.Sai Dilip Reddy.*
>
> On Thu, May 5, 2016 at 12:23 PM, Sai Dilip Reddy Kiralam <
> dkiralam@aadhya-analytics.com
> <javascript:_e(%7B%7D,'cvml','dkiralam@aadhya-analytics.com');>> wrote:
>
>>
>>
>>
>> *Best regards,*
>>
>> *K.Sai Dilip Reddy.*
>>
>> ---------- Forwarded message ----------
>> From: Spico Florin <spicoflorin@gmail.com
>> <javascript:_e(%7B%7D,'cvml','spicoflorin@gmail.com');>>
>> Date: Tue, May 3, 2016 at 1:02 PM
>> Subject: Re: Why tuples fail in spout
>> To: user@storm.apache.org
>> <javascript:_e(%7B%7D,'cvml','user@storm.apache.org');>
>>
>>
>> Hi!
>>   It could be that your process time for that tuples is greater than  topology.message.timeout.secs
>> 30 seconds. In these case have a look at your bolts processing time or how
>> fast you generate your data from spout that could not cope with your bolts.
>> You can:
>> 1. Increase your level your parallelism for your bolts.
>> 2. Throttling the messages from the spouts by setting out
>> topology.max.spout.pending
>> Please have a look at these questions:
>>
>> http://stackoverflow.com/questions/32322682/apache-storm-what-happens-to-a-tuple-when-no-bolts-are-available-to-consume-it
>>
>>
>> http://stackoverflow.com/questions/26536525/what-is-the-point-of-timing-out-tuples
>>
>> I hope that these help.
>>  Regards,
>>  Florin
>>
>> On Tue, May 3, 2016 at 9:42 AM, Sai Dilip Reddy Kiralam <
>> dkiralam@aadhya-analytics.com
>> <javascript:_e(%7B%7D,'cvml','dkiralam@aadhya-analytics.com');>> wrote:
>>
>>> but no failed count is shown for bolts
>>>
>>>
>>>
>>> *Best regards,*
>>>
>>> *K.Sai Dilip Reddy.*
>>>
>>> On Tue, May 3, 2016 at 11:19 AM, John Fang <xiaojian.fxj@alibaba-inc.com
>>> <javascript:_e(%7B%7D,'cvml','xiaojian.fxj@alibaba-inc.com');>> wrote:
>>>
>>>> Some tuples failed in the bolts. You can review the bolts' code. Maybe
>>>> your bolts' code trigger the fail() due to some reasons, Or the operation
>>>> of bolts need more time.
>>>>
>>>> ------------------------------------------------------------------
>>>> 发件人:Sai Dilip Reddy Kiralam <dkiralam@aadhya-analytics.com
>>>> <javascript:_e(%7B%7D,'cvml','dkiralam@aadhya-analytics.com');>>
>>>> 发送时间:2016年5月3日(星期二) 12:06
>>>> 收件人:user <user@storm.apache.org
>>>> <javascript:_e(%7B%7D,'cvml','user@storm.apache.org');>>
>>>> 主 题:Why tuples fail in spout
>>>>
>>>> Hi all,
>>>>
>>>>
>>>> I'm running storm topology in the local machine & storm UI shows some
>>>> tuples are failed in the spout.as per my knowledge spout tuples are
>>>> transferred to a bolts with out any failure.Can any of you help me out in
>>>> finding the reason of tuples failures in the spout.
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> *Best regards,*
>>>>
>>>> *K.Sai Dilip Reddy.*
>>>>
>>>>
>>>>
>>>
>>
>>
>

Mime
View raw message