storm-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Harsha <st...@harsha.io>
Subject Re: Storm Kafka Processing
Date Mon, 02 Feb 2015 17:00:40 GMT

Vineet, Can you try using the one in storm
https://github.com/apache/storm/tree/master/external/storm-kafka . This
is published into maven repo. So you can use the following <dependency>
<groupId>org.apache.storm</groupId> <artifactId>storm-kafka</artifactId>
<version>0.9.3</version> </dependency>

If you are using topic with partitions size 10 make sure you configured
your kafka spout with parallelism set to 10. Also make sure on the
producer side you are pushing data onto all of the 10 partitions so that
your kafka spout is fetching data from all of the 10 partitions. -Harsha


On Mon, Feb 2, 2015, at 08:55 AM, Vineet Mishra wrote:
> Hi Harsha,
>
> I am using storm.kafka.KafkaSpout.KafkaSpout implementation from
>
> https://github.com/wurstmeister/storm-kafka-0.8-plus
>
> Thanks!
>
> On Mon, Feb 2, 2015 at 8:14 PM, Harsha <storm@harsha.io> wrote:
>> __
>> Vineet, Which kafka spout are you using?
>>
>> -Harsha
>>
>>
>>
>> On Mon, Feb 2, 2015, at 05:25 AM, Vineet Mishra wrote:
>>> Hi,
>>>
>>> I am running Kafka Storm Engine to process real time data generated
>>> on a 3 node distributed cluster.
>>>
>>> Currently I have set 10 Executors for Storm Spout, which I don't
>>> think is running in parallel. Moreover earlier I was running the
>>> Kafka Topology with Replication Factor and Partitions as 1(which
>>> seems to have run comparatively faster), now I gave the Replication
>>> Factor as 3 and Partitions as 10 and I could see the performance
>>> degradation.
>>>
>>> Is there any way I can max utilize the available resource and get
>>> the max throughput of event processing.
>>>
>>> Looking for the expert suggestions at URGENT.
>>>
>>> Thanks!
>>
>


Mime
View raw message