storm-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Anandh Kumar <anandhz...@gmail.com>
Subject Re: Storm 0.10.0 Benchmark issue
Date Mon, 18 Apr 2016 05:56:10 GMT
Nikos,

Thanks for your reply.

I tried many value from low to high of max.spout.pending but not getting
throughput more than 7000/s.

I tried turning internal buffer but no improvement.

Did you tried any benchmark test on storm kafkaspout ? how much you got?
given the code so that I can run same test in my environment.

Regards,
-Anandh Kumar

On Sat, Apr 16, 2016 at 8:44 PM, Nikos R. Katsipoulakis <
nick.katsip@gmail.com> wrote:

> Hello Anandh,
>
> Why do you keep the max.spout.pending this low? I suggest increasing it to
> a higher value, because this might be the cause of your low throughput. In
> fact, it makes sense if you consider the following scenario:
>
> The KafkaSpout sends out 250 messages that try to fill out the Netty
> buffers before they are sent to the next node. When those are send, the
> Spout stops sending any additional tuples, until the ones "on-flight" are
> fully ACKed. Meanwhile, the Netty buffers are not full, therefore some time
> is spent up until they are sent out. Therefore, your throughput is
> implicitly "throttled"
>
> If I were you, I would keep the default values on the buffers (or even
> increase them a little - common practice for high throughput) and also try
> with a bigger value on max.spout.pending.
>
> Cheers,
> Nikos
>
> On Sat, Apr 16, 2016 at 11:06 AM, John Yost <hokiegeek2@gmail.com> wrote:
>
>> Hi Anandh,
>>
>> I recommend profiling your workers with something like jvisualvm to see
>> where your Bolt executors are spending their time.
>>
>> --John
>>
>> On Sat, Apr 16, 2016 at 10:01 AM, Anandh Kumar <anandhzedo@gmail.com>
>> wrote:
>>
>>> Hi Team,
>>>
>>> I benchmark the storm 0.10.0 with KafkaSpout and simple bolt not getting
>>>  throughput more than 7000 per seconds.
>>>
>>> Right now getting following throughput.
>>>
>>> Ack thread: 4
>>> max.spout.pending: 250
>>>
>>> With KafkaSpout with simple bolt
>>>
>>> Throughput - 7000/s message
>>>
>>> ======================================================
>>>
>>> Ack thread: 4
>>> max.spout.pending: 250
>>>
>>> With KafkaSpout with* no bolt *
>>>
>>> Throughput - 100000/s message
>>>
>>> ====================================================
>>>
>>> Why with bolt I am getting very less throughput?
>>>
>>> Not getting thoughput more than 7000/s. Tried to tune all the buffer
>>> parameters( send,receive buffer) not use.
>>>
>>> Please help me.
>>>
>>> Regards,
>>> -Anandh Kumar
>>>
>>
>>
>
>
> --
> Nikos R. Katsipoulakis,
> Department of Computer Science
> University of Pittsburgh
>

Mime
View raw message