storm-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From pradeep s <sreekumar.prad...@gmail.com>
Subject Re: Storm Spout failure
Date Sun, 07 Aug 2016 20:20:48 GMT
Adding the snippets from spout and bolt

fullMessage is the SQS message .

Spout

================

      spoutOutputCollector.emit(new Values(fullMessage, fullMessage
.getMessageId()),

                        fullMessage.getReceiptHandle());


DB Bolt

======

 outputCollector.emit(tuple, new Values(fullMessage));

    outputCollector.ack(tuple);


S3 Bolt

========

 outputCollector.emit(tuple, new Values(fullMessage));

    outputCollector.ack(tuple);


SQS Delete Bolt

============

 outputCollector.ack(tuple);




On Sun, Aug 7, 2016 at 1:06 PM, pradeep s <sreekumar.pradeep@gmail.com>
wrote:

> Time taken in bolts is very less . Total time of all bolts will come to
> less than 5 seconds only.
> Do i need to override the max spout pending param.
> If i process for small load, i dont see any issue.
>
> On Sat, Aug 6, 2016 at 11:20 AM, Andrew Montalenti <andrew@parsely.com>
> wrote:
>
>> Most commonly this is because your downstream components are failing to
>> ack your tuples. If a tuple goes unack'ed for the topology timeout period,
>> then it will fail at the spout.
>>
>> To debug this, make sure you get an ack for every tuple emitted from the
>> spout at every bolt processing stage. Also, you may want to double-check
>> your tuple anchor'ing -- this can sometimes be caused by improperly
>> anchored or unanchored tuples, as well.
>>
>> --
>> Andrew Montalenti | CTO, Parse.ly
>>
>> On Sat, Aug 6, 2016 at 1:22 PM, pradeep s <sreekumar.pradeep@gmail.com>
>> wrote:
>>
>>> Hi ,
>>> I am having a topology which reads messages from Amazon SQS and and
>>> there are three bolts .One bolt which writes to a RDS relational DB. Output
>>> of this is sent to a S3 write bolt.After this there is a SQS delete bolt
>>> which deletes the message from SQS.
>>> When i am testing a bigger load, i am seeing spout failures in Storm UI.
>>> But there are no failures in any of the bolts.Also no failure in log
>>> files.
>>> Any suggestion on the reason for spout failure and how to debug this.
>>> Topology timeout is set at default 30 secs.
>>> Thanks
>>> Pradeep S
>>>
>>
>>
>

Mime
View raw message