storm-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Nikos R. Katsipoulakis" <nick.kat...@gmail.com>
Subject Re: Storm 0.10.0 Benchmark issue
Date Mon, 18 Apr 2016 15:23:26 GMT
Hello again,

I am not using kafka-spouts and I did not run any benchmarks on my setup.
Can I ask what is your parallelism hint on the kafka-spouts? Because, it
might be the case that 7000 t/s might be the ceiling for your parallelism
hint and you may have to increase that in order to achieve higher
throughput.

Cheers,
Nikos

On Mon, Apr 18, 2016 at 1:56 AM, Anandh Kumar <anandhzedo@gmail.com> wrote:

> Nikos,
>
> Thanks for your reply.
>
> I tried many value from low to high of max.spout.pending but not getting
> throughput more than 7000/s.
>
> I tried turning internal buffer but no improvement.
>
> Did you tried any benchmark test on storm kafkaspout ? how much you got?
> given the code so that I can run same test in my environment.
>
> Regards,
> -Anandh Kumar
>
> On Sat, Apr 16, 2016 at 8:44 PM, Nikos R. Katsipoulakis <
> nick.katsip@gmail.com> wrote:
>
>> Hello Anandh,
>>
>> Why do you keep the max.spout.pending this low? I suggest increasing it
>> to a higher value, because this might be the cause of your low throughput.
>> In fact, it makes sense if you consider the following scenario:
>>
>> The KafkaSpout sends out 250 messages that try to fill out the Netty
>> buffers before they are sent to the next node. When those are send, the
>> Spout stops sending any additional tuples, until the ones "on-flight" are
>> fully ACKed. Meanwhile, the Netty buffers are not full, therefore some time
>> is spent up until they are sent out. Therefore, your throughput is
>> implicitly "throttled"
>>
>> If I were you, I would keep the default values on the buffers (or even
>> increase them a little - common practice for high throughput) and also try
>> with a bigger value on max.spout.pending.
>>
>> Cheers,
>> Nikos
>>
>> On Sat, Apr 16, 2016 at 11:06 AM, John Yost <hokiegeek2@gmail.com> wrote:
>>
>>> Hi Anandh,
>>>
>>> I recommend profiling your workers with something like jvisualvm to see
>>> where your Bolt executors are spending their time.
>>>
>>> --John
>>>
>>> On Sat, Apr 16, 2016 at 10:01 AM, Anandh Kumar <anandhzedo@gmail.com>
>>> wrote:
>>>
>>>> Hi Team,
>>>>
>>>> I benchmark the storm 0.10.0 with KafkaSpout and simple bolt not
>>>> getting  throughput more than 7000 per seconds.
>>>>
>>>> Right now getting following throughput.
>>>>
>>>> Ack thread: 4
>>>> max.spout.pending: 250
>>>>
>>>> With KafkaSpout with simple bolt
>>>>
>>>> Throughput - 7000/s message
>>>>
>>>> ======================================================
>>>>
>>>> Ack thread: 4
>>>> max.spout.pending: 250
>>>>
>>>> With KafkaSpout with* no bolt *
>>>>
>>>> Throughput - 100000/s message
>>>>
>>>> ====================================================
>>>>
>>>> Why with bolt I am getting very less throughput?
>>>>
>>>> Not getting thoughput more than 7000/s. Tried to tune all the buffer
>>>> parameters( send,receive buffer) not use.
>>>>
>>>> Please help me.
>>>>
>>>> Regards,
>>>> -Anandh Kumar
>>>>
>>>
>>>
>>
>>
>> --
>> Nikos R. Katsipoulakis,
>> Department of Computer Science
>> University of Pittsburgh
>>
>
>


-- 
Nikos R. Katsipoulakis,
Department of Computer Science
University of Pittsburgh

Mime
View raw message