storm-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Andrey Yegorov <andrey.yego...@gmail.com>
Subject Re: kafkaspout is very slow
Date Wed, 04 Feb 2015 21:19:29 GMT
have you tried increasing max spout pending parameter for the spout?

builder.setSpout("kafka",
                       new KafkaSpout(spoutConfig),
                       TOPOLOGY_NUM_TASKS_KAFKA_SPOUT)
          .setNumTasks(TOPOLOGY_NUM_TASKS_KAFKA_SPOUT)
          //the maximum parallelism you can have on a KafkaSpout is the
number of partitions
          .setMaxSpoutPending(*TOPOLOGY_MAX_SPOUT_PENDING*);

----------
Andrey Yegorov

On Tue, Feb 3, 2015 at 4:03 AM, clay teahouse <clayteahouse@gmail.com>
wrote:

> Hi all,
>
> In my topology,  kafka spout is responsible for over 85% of the latency. I
> have tried different spout max pending and played with the buffer size and
> fetch size, still no luck. Any hint on how to optimize the spout? The issue
> doesn't seem to be with the kafka side, as I see high throughput with the
> simple kafka consumer.
>
> thank you for your feedback
> Clay
>
>

Mime
View raw message