storm-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ravi Sharma <ping2r...@gmail.com>
Subject Re: Multiple Spouts in Same topology or Topology per spout
Date Sun, 11 Oct 2015 16:51:23 GMT
No 100% right ansers , u will have to test and see what will fit..

persoanlly i wud suggest Multiple spouts in one Topology and if you have N
node where topology will be running then each Spout(reading from one queue)
shud run N times in parallel.

if 2 Queues and say 4 Nodes
then one topolgy
4 Spouts reading from Queue1 in different nodes
4 spouts reading from Queue2 in different nodes

Ravi.

On Sun, Oct 11, 2015 at 5:25 PM, Abhishek priya <abhishek.priya@gmail.com>
wrote:

> I guess this is a question where there r no really correct answers. I'll
> certainly avoid#1 as it is better to keep logic separate and lightweight.
>
> If your downstream bolts are same, then it makes senses to keep them in
> same topology but if they r totally different, I'll keep them in two
> different topologies. That will allow me to independently deploy and scale
> the topology. But if the rest of logic is same I topology scaling and
> resource utilization will be better with one topology.
>
> I hope this helps..
>
> Sent somehow....
>
> > On Oct 11, 2015, at 9:07 AM, Ankur Garg <ankurgarg9@gmail.com> wrote:
> >
> > Hi ,
> >
> > So I have a situation where I want to read messages from different
> queues hosted in a Rabbitmq Server .
> >
> > Now , there are three ways which I can think to leverage Apache Storm
> here :-
> >
> > 1) Use the same Spout (say Spout A) to read messages from different
> queues and based on the messages received emit it to different Bolts.
> >
> > 2) Use different Spout (Spout A and Spout B and so on) within the same
> topology (say Topology A) to read messages from different queues .
> >
> > 3) Use Different Spouts one within eachTopology (Topology A , Topology B
> and so on) to read messages from different queues .
> >
> > Which is the best way to process this considering I want high throughput
> (more no of queue messages to be processed concurrently) .
> >
> > Also , If In use same Topology for all Spouts (currently though
> requirement is for 2 spouts)  will failure in one Spout (or its associated
> Bolts) effect the second or will they both continue working separately even
> if some failure is in Spout B ?
> >
> > Cost wise , how much would it be to maintain two different topologies .
> >
> > Looking for inputs from members here.
> >
> > Thanks
> > Ankur
> >
> >
>

Mime
View raw message