storm-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ankur Garg <ankurga...@gmail.com>
Subject Re: Multiple Spouts in Same topology or Topology per spout
Date Mon, 12 Oct 2015 08:50:27 GMT
But what if MongoDb bolt has some error , in that case I suppose the entire
tuple will be replayed from Spout meaning it will have to redo the
operation of inserting into sql . Is there a way I can skip inserting into
mysql ?

On Mon, Oct 12, 2015 at 1:54 PM, Susheel Kumar Gadalay <skgadalay@gmail.com>
wrote:

> It is better to have 2 bolts - mysql bolt and mongodb bolt.
>
> Let mysql bolt forward the tuple to mongodb bolt, so in case of error
> it won't  emit.
>
> On 10/12/15, Ankur Garg <ankurgarg9@gmail.com> wrote:
> > So I have a situation where the tuple received on Spout has to be saved
> to
> > mysql database and mongoDb as well .
> >
> > What should be better . Using 1 bolt to save it into mysql and MongoDb
> or 2
> > seperate Bolts (One for saving into mysql and other for saving into
> Mongo).
> >
> > What happens when the exception occurs while saving into mysql ? I
> believe
> > I will get acknowledgement inside the fail method in my Spout . So If I
> > reprocess it using 2 bolts , I believe it will again be sent to Bolt for
> > saving into Mongo database .
> >
> > If the above is true , will having 2 seperate bolts be of any advantage ?
> > how can I configure things so that Failure while inserting into mysql
> does
> > not impact inserting into MongoDb .
> >
> > Thanks
> > Ankur
> >
> > On Sun, Oct 11, 2015 at 10:57 PM, Ravi Sharma <ping2ravi@gmail.com>
> wrote:
> >
> >> That depends if ur spout error has affected jvm or normal application
> >> error
> >>
> >> performance issue in case of lot of errors, I don't think there is any
> >> issue be coz of errors themselves but ofcourse if u r retrying these
> >> messages on failure then that means u will be processing lot of messages
> >> then normal and overall throughput will go down
> >>
> >> Ravi
> >>
> >> If ur topology has enabled acknowledgment that means spout will always
> >> receive
> >> On 11 Oct 2015 18:15, "Ankur Garg" <ankurgarg9@gmail.com> wrote:
> >>
> >>>
> >>> Thanks for the reply Abhishek and Ravi .
> >>>
> >>> One question though , going with One topology with multiple spouts
> >>> ...What if something goes wrong in One spout or its associated bolts ..
> >>> Does it impact other Spout as well?
> >>>
> >>> Thanks
> >>> Ankur
> >>>
> >>> On Sun, Oct 11, 2015 at 10:21 PM, Ravi Sharma <ping2ravi@gmail.com>
> >>> wrote:
> >>>
> >>>> No 100% right ansers , u will have to test and see what will fit..
> >>>>
> >>>> persoanlly i wud suggest Multiple spouts in one Topology and if you
> >>>> have
> >>>> N node where topology will be running then each Spout(reading from one
> >>>> queue) shud run N times in parallel.
> >>>>
> >>>> if 2 Queues and say 4 Nodes
> >>>> then one topolgy
> >>>> 4 Spouts reading from Queue1 in different nodes
> >>>> 4 spouts reading from Queue2 in different nodes
> >>>>
> >>>> Ravi.
> >>>>
> >>>> On Sun, Oct 11, 2015 at 5:25 PM, Abhishek priya <
> >>>> abhishek.priya@gmail.com> wrote:
> >>>>
> >>>>> I guess this is a question where there r no really correct answers.
> >>>>> I'll certainly avoid#1 as it is better to keep logic separate and
> >>>>> lightweight.
> >>>>>
> >>>>> If your downstream bolts are same, then it makes senses to keep
them
> >>>>> in
> >>>>> same topology but if they r totally different, I'll keep them in
two
> >>>>> different topologies. That will allow me to independently deploy
and
> >>>>> scale
> >>>>> the topology. But if the rest of logic is same I topology scaling
and
> >>>>> resource utilization will be better with one topology.
> >>>>>
> >>>>> I hope this helps..
> >>>>>
> >>>>> Sent somehow....
> >>>>>
> >>>>> > On Oct 11, 2015, at 9:07 AM, Ankur Garg <ankurgarg9@gmail.com>
> >>>>> > wrote:
> >>>>> >
> >>>>> > Hi ,
> >>>>> >
> >>>>> > So I have a situation where I want to read messages from different
> >>>>> queues hosted in a Rabbitmq Server .
> >>>>> >
> >>>>> > Now , there are three ways which I can think to leverage Apache
> >>>>> > Storm
> >>>>> here :-
> >>>>> >
> >>>>> > 1) Use the same Spout (say Spout A) to read messages from different
> >>>>> queues and based on the messages received emit it to different Bolts.
> >>>>> >
> >>>>> > 2) Use different Spout (Spout A and Spout B and so on) within
the
> >>>>> same topology (say Topology A) to read messages from different queues
> >>>>> .
> >>>>> >
> >>>>> > 3) Use Different Spouts one within eachTopology (Topology A
,
> >>>>> Topology B and so on) to read messages from different queues .
> >>>>> >
> >>>>> > Which is the best way to process this considering I want high
> >>>>> throughput (more no of queue messages to be processed concurrently)
.
> >>>>> >
> >>>>> > Also , If In use same Topology for all Spouts (currently though
> >>>>> requirement is for 2 spouts)  will failure in one Spout (or its
> >>>>> associated
> >>>>> Bolts) effect the second or will they both continue working
> separately
> >>>>> even
> >>>>> if some failure is in Spout B ?
> >>>>> >
> >>>>> > Cost wise , how much would it be to maintain two different
> >>>>> > topologies
> >>>>> .
> >>>>> >
> >>>>> > Looking for inputs from members here.
> >>>>> >
> >>>>> > Thanks
> >>>>> > Ankur
> >>>>> >
> >>>>> >
> >>>>>
> >>>>
> >>>>
> >>>
> >
>

Mime
View raw message