storm-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Arun Mahadevan <ar...@apache.org>
Subject Re: Behavior of Storm when buffers fill
Date Tue, 26 Sep 2017 16:05:18 GMT
If Bolt B’s receive queue is full the tuple will be put into an overflow buffer. If  back
pressure is not enabled it will keep on filling the overflow buffer and eventually cause an
OOM.

You might also want to see the proposed changes in STORM-2306 where the receive queues are
going to be bounded and the back pressure is allowed to propagate backwards up to the spout
at which point the framework would stop invoking the nextTuple.

Thanks,
Arun

From:  Lawrence Craft <lawrence.craft@gmail.com>
Reply-To:  "user@storm.apache.org" <user@storm.apache.org>
Date:  Tuesday, September 26, 2017 at 2:01 PM
To:  "user@storm.apache.org" <user@storm.apache.org>
Subject:  Behavior of Storm when buffers fill

Hi all, 

I'm trying to get a better understanding of how Storm behaves in adverse conditions. I read
http://www.michael-noll.com/blog/2013/06/21/understanding-storm-internal-message-buffers/,
describing the various buffers between different workers and components.

My question: if bolt A emits a tuple that is routed to Bolt B, but Bolt B's executor buffer
is full, how does the system react? I would guess it'd be kept in the worker's receive buffer,
but what happens when that fills? Does Bolt A's worker send thread block and stall?

Obviously this is not an ideal situation and buffers/backpressure watermarks should be set
precisely to avoid something like this. I'd just like to get an idea of what happens.

Lawrence


Mime
View raw message