storm-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Nikos R. Katsipoulakis" <nick.kat...@gmail.com>
Subject Re: Storm 0.10.0 Benchmark issue
Date Sat, 16 Apr 2016 15:14:01 GMT
Hello Anandh,

Why do you keep the max.spout.pending this low? I suggest increasing it to
a higher value, because this might be the cause of your low throughput. In
fact, it makes sense if you consider the following scenario:

The KafkaSpout sends out 250 messages that try to fill out the Netty
buffers before they are sent to the next node. When those are send, the
Spout stops sending any additional tuples, until the ones "on-flight" are
fully ACKed. Meanwhile, the Netty buffers are not full, therefore some time
is spent up until they are sent out. Therefore, your throughput is
implicitly "throttled"

If I were you, I would keep the default values on the buffers (or even
increase them a little - common practice for high throughput) and also try
with a bigger value on max.spout.pending.

Cheers,
Nikos

On Sat, Apr 16, 2016 at 11:06 AM, John Yost <hokiegeek2@gmail.com> wrote:

> Hi Anandh,
>
> I recommend profiling your workers with something like jvisualvm to see
> where your Bolt executors are spending their time.
>
> --John
>
> On Sat, Apr 16, 2016 at 10:01 AM, Anandh Kumar <anandhzedo@gmail.com>
> wrote:
>
>> Hi Team,
>>
>> I benchmark the storm 0.10.0 with KafkaSpout and simple bolt not getting
>>  throughput more than 7000 per seconds.
>>
>> Right now getting following throughput.
>>
>> Ack thread: 4
>> max.spout.pending: 250
>>
>> With KafkaSpout with simple bolt
>>
>> Throughput - 7000/s message
>>
>> ======================================================
>>
>> Ack thread: 4
>> max.spout.pending: 250
>>
>> With KafkaSpout with* no bolt *
>>
>> Throughput - 100000/s message
>>
>> ====================================================
>>
>> Why with bolt I am getting very less throughput?
>>
>> Not getting thoughput more than 7000/s. Tried to tune all the buffer
>> parameters( send,receive buffer) not use.
>>
>> Please help me.
>>
>> Regards,
>> -Anandh Kumar
>>
>
>


-- 
Nikos R. Katsipoulakis,
Department of Computer Science
University of Pittsburgh

Mime
View raw message