have you tried increasing max spout pending parameter for the spout?

builder.setSpout("kafka",
                       new KafkaSpout(spoutConfig),
                       TOPOLOGY_NUM_TASKS_KAFKA_SPOUT)
          .setNumTasks(TOPOLOGY_NUM_TASKS_KAFKA_SPOUT)
          //the maximum parallelism you can have on a KafkaSpout is the number of partitions
          .setMaxSpoutPending(TOPOLOGY_MAX_SPOUT_PENDING);

----------
Andrey Yegorov

On Tue, Feb 3, 2015 at 4:03 AM, clay teahouse <clayteahouse@gmail.com> wrote:
Hi all,

In my topology,  kafka spout is responsible for over 85% of the latency. I have tried different spout max pending and played with the buffer size and fetch size, still no luck. Any hint on how to optimize the spout? The issue doesn't seem to be with the kafka side, as I see high throughput with the simple kafka consumer.

thank you for your feedback
Clay