storm-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Anandh Kumar <anandhz...@gmail.com>
Subject Re: Storm 0.10.0 Benchmark issue
Date Tue, 19 Apr 2016 07:31:47 GMT
Hi Nikos,

My topic have 6 partition so my kafka-spouts parallelism hint also 6.

My worker is 10
KafkaSpout executor is 6

Only with Kafka-spout without bolt I got 1,00,000/s but excepted 10,00,000/s

Regards,
-Anandh Kumar

On Mon, Apr 18, 2016 at 8:53 PM, Nikos R. Katsipoulakis <
nick.katsip@gmail.com> wrote:

> Hello again,
>
> I am not using kafka-spouts and I did not run any benchmarks on my setup.
> Can I ask what is your parallelism hint on the kafka-spouts? Because, it
> might be the case that 7000 t/s might be the ceiling for your parallelism
> hint and you may have to increase that in order to achieve higher
> throughput.
>
> Cheers,
> Nikos
>
> On Mon, Apr 18, 2016 at 1:56 AM, Anandh Kumar <anandhzedo@gmail.com>
> wrote:
>
>> Nikos,
>>
>> Thanks for your reply.
>>
>> I tried many value from low to high of max.spout.pending but not getting
>> throughput more than 7000/s.
>>
>> I tried turning internal buffer but no improvement.
>>
>> Did you tried any benchmark test on storm kafkaspout ? how much you got?
>> given the code so that I can run same test in my environment.
>>
>> Regards,
>> -Anandh Kumar
>>
>> On Sat, Apr 16, 2016 at 8:44 PM, Nikos R. Katsipoulakis <
>> nick.katsip@gmail.com> wrote:
>>
>>> Hello Anandh,
>>>
>>> Why do you keep the max.spout.pending this low? I suggest increasing it
>>> to a higher value, because this might be the cause of your low throughput.
>>> In fact, it makes sense if you consider the following scenario:
>>>
>>> The KafkaSpout sends out 250 messages that try to fill out the Netty
>>> buffers before they are sent to the next node. When those are send, the
>>> Spout stops sending any additional tuples, until the ones "on-flight" are
>>> fully ACKed. Meanwhile, the Netty buffers are not full, therefore some time
>>> is spent up until they are sent out. Therefore, your throughput is
>>> implicitly "throttled"
>>>
>>> If I were you, I would keep the default values on the buffers (or even
>>> increase them a little - common practice for high throughput) and also try
>>> with a bigger value on max.spout.pending.
>>>
>>> Cheers,
>>> Nikos
>>>
>>> On Sat, Apr 16, 2016 at 11:06 AM, John Yost <hokiegeek2@gmail.com>
>>> wrote:
>>>
>>>> Hi Anandh,
>>>>
>>>> I recommend profiling your workers with something like jvisualvm to see
>>>> where your Bolt executors are spending their time.
>>>>
>>>> --John
>>>>
>>>> On Sat, Apr 16, 2016 at 10:01 AM, Anandh Kumar <anandhzedo@gmail.com>
>>>> wrote:
>>>>
>>>>> Hi Team,
>>>>>
>>>>> I benchmark the storm 0.10.0 with KafkaSpout and simple bolt not
>>>>> getting  throughput more than 7000 per seconds.
>>>>>
>>>>> Right now getting following throughput.
>>>>>
>>>>> Ack thread: 4
>>>>> max.spout.pending: 250
>>>>>
>>>>> With KafkaSpout with simple bolt
>>>>>
>>>>> Throughput - 7000/s message
>>>>>
>>>>> ======================================================
>>>>>
>>>>> Ack thread: 4
>>>>> max.spout.pending: 250
>>>>>
>>>>> With KafkaSpout with* no bolt *
>>>>>
>>>>> Throughput - 100000/s message
>>>>>
>>>>> ====================================================
>>>>>
>>>>> Why with bolt I am getting very less throughput?
>>>>>
>>>>> Not getting thoughput more than 7000/s. Tried to tune all the buffer
>>>>> parameters( send,receive buffer) not use.
>>>>>
>>>>> Please help me.
>>>>>
>>>>> Regards,
>>>>> -Anandh Kumar
>>>>>
>>>>
>>>>
>>>
>>>
>>> --
>>> Nikos R. Katsipoulakis,
>>> Department of Computer Science
>>> University of Pittsburgh
>>>
>>
>>
>
>
> --
> Nikos R. Katsipoulakis,
> Department of Computer Science
> University of Pittsburgh
>

Mime
View raw message