storm-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From clay teahouse <clayteaho...@gmail.com>
Subject Re: kafkaspout is very slow
Date Wed, 04 Feb 2015 21:58:06 GMT
 I have this issue with any amount of load. Different max spout pendings do
not seem to make much a difference. I've lowered this parameter to 100,
still a little difference . At this point the bolt consuming the data does
no processing.

On Wed, Feb 4, 2015 at 3:26 PM, Haralds Ulmanis <haralds@evilezh.net> wrote:

> I'm not sure, that i understand your problem .. but here is few points:
> If you have large pending spout size and slow processing - you will see
> large latency at kafka spout probably. Spout emits message .. it stays in
> queue for long time (that will add latency) .. and finally is processed and
> ack received. You will see queue time + processing time in kafka spout
> latency.
> Take a look at load factors of your bolts - are they close to 1 or more ?
> and load factor of kafka spout.
>
> On 4 February 2015 at 21:19, Andrey Yegorov <andrey.yegorov@gmail.com>
> wrote:
>
>> have you tried increasing max spout pending parameter for the spout?
>>
>> builder.setSpout("kafka",
>>                        new KafkaSpout(spoutConfig),
>>                        TOPOLOGY_NUM_TASKS_KAFKA_SPOUT)
>>           .setNumTasks(TOPOLOGY_NUM_TASKS_KAFKA_SPOUT)
>>           //the maximum parallelism you can have on a KafkaSpout is the
>> number of partitions
>>           .setMaxSpoutPending(*TOPOLOGY_MAX_SPOUT_PENDING*);
>>
>> ----------
>> Andrey Yegorov
>>
>> On Tue, Feb 3, 2015 at 4:03 AM, clay teahouse <clayteahouse@gmail.com>
>> wrote:
>>
>>> Hi all,
>>>
>>> In my topology,  kafka spout is responsible for over 85% of the latency.
>>> I have tried different spout max pending and played with the buffer size
>>> and fetch size, still no luck. Any hint on how to optimize the spout? The
>>> issue doesn't seem to be with the kafka side, as I see high throughput with
>>> the simple kafka consumer.
>>>
>>> thank you for your feedback
>>> Clay
>>>
>>>
>>
>

Mime
View raw message