storm-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Patrick.Brinton <Patrick.Brin...@target.com>
Subject Re: Distribute load across multiple storm clusters.
Date Fri, 29 Apr 2016 05:10:25 GMT
Darsh,
I am in a bit of a crush for a deployment and perf testing but I will ask my experts to take
a look tomorrow.  I will set up a single nimbus and 3 zookeepers in one data center, then
I will distribute the supervisors and see what happens.  I think we will always be limited
in how we back up nimbus but I think we should be able to share processing for supervisors
across data centers.

Keep in touch and let me know if this is really what you are looking to test.  I have a lot
of toys at the moment and this seems like a worth while test for everyone.

Patrick


Patrick Brinton Sr. Engineer SWLM | ┬ĄTarget | 7000 Target Parkway North | Brooklyn Park,
MN  55445 | 612.599.6523 (ph) | Patrick.Brinton@target.com<mailto:C3@target.com>

From: Darsh <darsh.patil@gmail.com<mailto:darsh.patil@gmail.com>>
Reply-To: "user@storm.apache.org<mailto:user@storm.apache.org>" <user@storm.apache.org<mailto:user@storm.apache.org>>
Date: Thursday, April 28, 2016 at 5:29 PM
To: "user@storm.apache.org<mailto:user@storm.apache.org>" <user@storm.apache.org<mailto:user@storm.apache.org>>
Subject: Re: Distribute load across multiple storm clusters.
Patrick,

Thank you for replying. I did try but load isn't distributed. Both clusters are processing
all the events individually on the topic.


Darsh

On Thu, Apr 28, 2016 at 2:56 PM, Patrick.Brinton <Patrick.Brinton@target.com<mailto:Patrick.Brinton@target.com>>
wrote:
Darsh,
I have never tried but I have a setup where I could try.  As long as you point to the same
zooKeeper I think it would work.  Give it a try and let me know if you hit issues.  If you
do I will mimic your setup and we should be able to figure it out.

Patrick


Patrick Brinton Sr. Engineer SWLM | ┬ĄTarget | 7000 Target Parkway North | Brooklyn Park,
MN  55445 | 612.599.6523<tel:612.599.6523> (ph) | Patrick.Brinton@target.com<mailto:C3@target.com>

From: Darsh <darsh.patil@gmail.com<mailto:darsh.patil@gmail.com>>
Reply-To: "user@storm.apache.org<mailto:user@storm.apache.org>" <user@storm.apache.org<mailto:user@storm.apache.org>>
Date: Thursday, April 28, 2016 at 12:39 PM
To: "user@storm.apache.org<mailto:user@storm.apache.org>" <user@storm.apache.org<mailto:user@storm.apache.org>>
Subject: Distribute load across multiple storm clusters.

Hi,


Is it possible to to distribute load across 2 clusters if I deploy same topology to 2 storm
clusters with same spout id?


We have 2 separate storm clusters with 0.10.0 version of storm running in 2 different availability
zones. We are using storm-kafka spout (with simple consumer) to process data from kafka and
external common zookeeper to store the kafka offsets.Kafka Topic has  32 partitions. 8  executors(parallelism_hint)
for spout in each cluster.






Thanks

Darsh



--
Thanks

Darsh
Mime
View raw message