storm-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Darsh <darsh.pa...@gmail.com>
Subject Re: Distribute load across multiple storm clusters.
Date Fri, 29 Apr 2016 18:42:27 GMT
Hi Patrick,

Sorry I think I confused you with my setup. I have separate nimbus for each
storm cluster.

Setup I used for testing,

3 node zk cluster with one node in 3 availability zones.

storm cluster 1=nimbus and 3 supervisor in availability zone 1

storm cluster 2=nimbus and 3 supervisor in availability zone 2



Darsh


On Thu, Apr 28, 2016 at 10:10 PM, Patrick.Brinton <
Patrick.Brinton@target.com> wrote:

> Darsh,
> I am in a bit of a crush for a deployment and perf testing but I will ask
> my experts to take a look tomorrow.  I will set up a single nimbus and 3
> zookeepers in one data center, then I will distribute the supervisors and
> see what happens.  I think we will always be limited in how we back up
> nimbus but I think we should be able to share processing for supervisors
> across data centers.
>
> Keep in touch and let me know if this is really what you are looking to
> test.  I have a lot of toys at the moment and this seems like a worth while
> test for everyone.
>
> Patrick
>
> Patrick Brinton Sr. Engineer SWLM | ┬ĄTarget | 7000 Target Parkway North
> | Brooklyn Park, MN  55445 | 612.599.6523* (ph) *|
> Patrick.Brinton@target.com <C3@target.com>
>
> From: Darsh <darsh.patil@gmail.com>
> Reply-To: "user@storm.apache.org" <user@storm.apache.org>
> Date: Thursday, April 28, 2016 at 5:29 PM
> To: "user@storm.apache.org" <user@storm.apache.org>
> Subject: Re: Distribute load across multiple storm clusters.
> Patrick,
>
> Thank you for replying. I did try but load isn't distributed. Both
> clusters are processing all the events individually on the topic.
>
>
> Darsh
>
> On Thu, Apr 28, 2016 at 2:56 PM, Patrick.Brinton <
> Patrick.Brinton@target.com> wrote:
>
>> Darsh,
>> I have never tried but I have a setup where I could try.  As long as you
>> point to the same zooKeeper I think it would work.  Give it a try and let
>> me know if you hit issues.  If you do I will mimic your setup and we should
>> be able to figure it out.
>>
>> Patrick
>>
>> Patrick Brinton Sr. Engineer SWLM | ┬ĄTarget | 7000 Target Parkway North
>> | Brooklyn Park, MN  55445 | 612.599.6523* (ph) *|
>> Patrick.Brinton@target.com <C3@target.com>
>>
>> From: Darsh <darsh.patil@gmail.com>
>> Reply-To: "user@storm.apache.org" <user@storm.apache.org>
>> Date: Thursday, April 28, 2016 at 12:39 PM
>> To: "user@storm.apache.org" <user@storm.apache.org>
>> Subject: Distribute load across multiple storm clusters.
>>
>> Hi,
>>
>>
>> Is it possible to to distribute load across 2 clusters if I deploy same
>> topology to 2 storm clusters with same spout id?
>>
>>
>> We have 2 separate storm clusters with 0.10.0 version of storm running in
>> 2 different availability zones. We are using storm-kafka spout (with simple
>> consumer) to process data from kafka and external common zookeeper to store
>> the kafka offsets.Kafka Topic has  32 partitions. 8  executors(
>> *parallelism_hint*) for spout in each cluster.
>>
>>
>>
>>
>>
>>
>> Thanks
>>
>> Darsh
>>
>
>
>
> --
> Thanks
>
> Darsh
>



-- 
Thanks

Darsh

Mime
View raw message