storm-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ganesh Chandrasekaran <gchandraseka...@wayfair.com>
Subject Re: AMQP spout gets stuck
Date Sat, 20 Aug 2016 12:21:02 GMT
I have experienced a similar issue with AMQP on 0.9.3 and it was related to some netty issue
which were solved on 0.9.4. We upgraded to that version and it fixed the issue. I would highly
recommend using 0.9.4 (or 0.9.6 which is my latest version that I am using with AMQP). It
has been stable ever since.
________________________________
From: Cody Lee <codylee@wellaware.us>
Sent: Tuesday, August 16, 2016 9:07:16 AM
To: user@storm.apache.org
Subject: Re: AMQP spout gets stuck

We're on 1.x here, but I might be able to help. Taking off dev@

Topology/Storm questions:

1.       The executors have been healthy and haven't been erroring out?

2.       An appropriate tuple timeout has been set for the topology? (This will fail a "stuck"
tuple and keep things moving, but view it more as a bandaid and not solution. This triggering
indicates that tuples are not getting acked or failed in time, or the setting is just too
small).

Code-Specific

1.       Are you ensuring that you fully ack or fail every tuple in the tuple tree coming
from this spout? It's easy to miss, especially under edge-cases (I've been guilty of this
myself before).  Also the collector is not thread safe, be sure to only ack or fail inside
the bolt thread if you're doing anything async.

2.       Are there dependency failures that are holding up processing?

3.       Incoming messages are as expected, ie no unexpected message drift?

4.       Are you using any windowing functions? If so, is it possible that some executor windows
are able to fill up to a trigger point to execute?

Cody

From: Rahul Dahiya <rahul.dahiya@impetus.co.in>
Reply-To: "user@storm.apache.org" <user@storm.apache.org>
Date: Tuesday, August 16, 2016 at 6:52 AM
To: "dev@storm.apache.org" <dev@storm.apache.org>, "user@storm.apache.org" <user@storm.apache.org>
Subject: AMQP spout gets stuck

Hi Team,



We are using apache-storm v 0.9,3 to read messages from RMQ using AMQP spout and processing
the messages to save in the database.



Recently we have started facing an issue in the topology where the messages gets stuck in
unacked stated in the RMQ and we see failures in the AMQP spout itself. To solve this we have
to redeploy that specific topology . It works fine for sometime and process messages at a
normal rate. After some time (not specific interval ) the messages gets stuck in unacked state
in RMQ and we have to repeat the process all over again.



Also, sometimes it runs flawlessly throughout the day and then out of the blue it starts experiencing
the issue.

We haven't done any code change in months and we tried running the topology in a separate
cluster (RMQ and Storm) as well with no success .



Also there are multiple topologies deployed on the storm cluster and most of them run perfectly
fine (Although we have seen issues with some but they got resolved).



We are using Amazon cloud for the cluster.



Any help regarding this is highly appreciated. Thanks a ton in advance.



Regards,

Rahul

________________________________






NOTE: This message may contain information that is confidential, proprietary, privileged or
otherwise protected by law. The message is intended solely for the named addressee. If received
in error, please destroy and notify the sender. Any use of this email is prohibited when received
in error. Impetus does not represent, warrant and/or guarantee, that the integrity of this
communication has been maintained nor that the communication is free of errors, virus, interception
or interference.

Mime
View raw message