storm-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Roshan Naik <roshan_n...@yahoo.com>
Subject Re: Re:Flooded topology after a full GC
Date Fri, 20 Dec 2019 22:35:14 GMT


1) To disable ACKing just set ACKer bolts instances (i.e topology.acker.executors)  = 0.
Otherwise it is enabled. For ACKing/replay to work correctly you need the anchoring &
acking to occur like you described. Unanchored tuples will not participate in completion tracking
& replay. See https://storm.apache.org/releases/current/Guaranteeing-message-processing.html


2) When ingestion rate remains higher than drain rate.... In Storm 1.x, internal buffers will
overflow and workers will crash & restart repeatedly IF no form of backpressure (BP) is
enabled. It has two ways to get BP.  One way is via topology.max.spout.pending which works
only if ACKing is enabled. Alternatively there is another ZK based BP mechanism that can be
enabled (i think disabled by default).  If worker crashes and ACKing is disabled, the messages
currently contained in the worker are lost. If ACKing is enabled, timeouts will occur on the
tuple tree and the spout will remit the root the tuple tree.

In Storm v2.0, there is a better BP model integrated into the messaging subsystem which cannot
be disabled and doesn't use ZK. So v2.x wont crash due in any BP scenario. It also retains
the topology.max.spout.pending. (https://issues.apache.org/jira/browse/STORM-2306) 




Mime
View raw message