storm-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Anandh Kumar <anandhz...@gmail.com>
Subject Re: Losing Tuples
Date Tue, 19 Apr 2016 11:27:21 GMT
Johnson,

I am facing some other problem can you please help me.

I done the bechmarked testing on storm with KafkaSpout and simple bolt I
got 10000 message per secs.

I tried turning max.spout.pending with 20000 and tuning topology internal
buffer not getting more than this.

Configuration:

KafkaSpout - 6 executer
Bolt - 6 executer
max.spout pending - 20000

Can you please tell me what are the configuration you have and what is your
throughput?

On Mon, Apr 18, 2016 at 11:39 PM, <johnson_d4221@yahoo.com> wrote:

> The spout is a KafkaSpout and I only have one spout task.
>
> The reason I set the MaxSpoutPendingValue so high was that in the
> topology, each tuple processed in a bolt tends to create more tuples. So,
> although the KafkaSpout only receives one message, it results in thousands
> of tuples downstream.
>
>
> On Sunday, April 17, 2016 6:56 AM, Kevin Conaway <
> kevin.a.conaway@gmail.com> wrote:
>
>
> What type of spout is it? How many spout tasks do you have?
>
> Maxspoutpending seems pretty high so its possible the tuples could be
> toming out in the queue and if the spout isnt reliable, or if acking is
> disabled, they will be discarded
>
> On Friday, April 15, 2016, <johnson_d4221@yahoo.com> wrote:
>
> I've placed two logs in the bolts to verify that tuples are missing. One
> log right before the tuple is emitted and another at the beginning of the
> execute method for the downstream bolt. These logs should contain the same
> statements, however, the downstream bolt is lacking close to 1000 of the
> 21,000 tuples it should be receiving.
>
>
> On Friday, April 15, 2016 12:56 PM, Kevin Conaway <
> kevin.a.conaway@gmail.com> wrote:
>
>
> How are you verifying that the tuples are failing?  If you're looking at
> storm UI for the exact counts you may be mislead.  Storm samples tuples for
> at a configurable rate (defaulted to 0.05) and extrapolates the metrics
> shown in the UI based on that number.  For dev or testing purposes you can
> set the _topology.stats.sample.rate_ to 1 in storm.yaml which will cause
> storm to compute stats based on every tuple.
>
> On Fri, Apr 15, 2016 at 12:33 PM, <johnson_d4221@yahoo.com> wrote:
>
> Hi all,
>
> I've recently run into a problem where my topology seems to be losing
> tuples after some continuous processing. That is the number of tuples
> emitted from one bolt doesn't equal the number of tuples ack'ed for the
> downstream bolt. It's also not reporting any tuples as having failed, I ack
> immediately in each exectue method, and there seem to be no errors in the
> logs. Due to the nature of the topology, one bolt tends to emit about 10
> tuples for each tuple that it receives, resulting in the topology itself
> getting backed up relatively quickly. I've read in other articles that this
> can result in a memory leak, which might be the cause of my lost tuples.
>
> My question is what configuration properties of the topology can I change
> that would potentially resolve this problem? I currently have my
> executor.send.buffer and executor.receive.buffer set at 16384, the
> maxSpoutPending at 500000, and the tupleTimeout at 300000, which I thought
> would help, but still have not seen any improvement. Or is there something
> else that might be causing this problem?
>
> Thanks
>
>
>
>
> --
> Kevin Conaway
> http://www.linkedin.com/pub/kevin-conaway/7/107/580/
> https://github.com/kevinconaway
>
>
>
>
> --
> Kevin Conaway
> http://www.linkedin.com/pub/kevin-conaway/7/107/580/
> https://github.com/kevinconaway
>
>
>
>

Mime
View raw message