storm-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ravi Sharma <ping2r...@gmail.com>
Subject Re: Multiple Spouts in Same topology or Topology per spout
Date Mon, 12 Oct 2015 09:52:58 GMT
Hi Ankur,

Storm's design is stateless, so storm cant store any info about what bolts
were successful and which one failed.
Idea is to replay the message again without affecting the final outcome.
(means if mysql was success, it shudnt add two rows in case its replayed)

>From looking at far i would say you may be fixing an issue which hasnt
happened yet. Assumption is that one DB will be failing a lot, i guess this
may not be real case.
Any of the DB can fail once in a while and replaying them shudnt affect ur
performance. (say less then 10% Message failed) , you will be planning
atleast 50% more capacity then ur max load.


If you really want it to be very effective, i say use something like redis
and store your bolt status with message id there, so every time you plan to
start a bolt proessing check if you have already completed it succesfully,
if yes then skip it.
I have defined my own MessageId object and always put a retry count in it.
So first one goes with 0, and at that moment you can avoid the redis/nosql
checks.
But then u r adding one more technology and it just increased the
complexity.


Whatever design you choose, i will still suggest to use two bolts, Monogo
and mysql both are different cluster(hardware) and technology(software),
they both will have different throughput and scalability. And as per your
requirment you dont care if data hasnt reached to one exactly at same time,
no atomicity (basically its not one transaction), so you dont want to slow
down one system because other is slower.


Last suggestion is to go with two spouts....  both will read from same
topic(not queue), so all messages will be delivered to both Spouts. One
Spout will send message to Mysql Bolt other will send to Mongo Bolt.


Ravi.




Ravi.













On Mon, Oct 12, 2015 at 10:14 AM, Ankur Garg <ankurgarg9@gmail.com> wrote:

> LOL .. I was looking for something better :) ..If you see then having
> multiple bolts here do not help much .. It would have helped had there been
> a provision to skip the already executed Bolts .
>
>
> I believe this should be there in Storm .
>
> Thanks
> Ankur
>
> On Mon, Oct 12, 2015 at 2:42 PM, Susheel Kumar Gadalay <
> skgadalay@gmail.com> wrote:
>
>> Check and insert
>>
>> On 10/12/15, Ankur Garg <ankurgarg9@gmail.com> wrote:
>> > But what if MongoDb bolt has some error , in that case I suppose the
>> entire
>> > tuple will be replayed from Spout meaning it will have to redo the
>> > operation of inserting into sql . Is there a way I can skip inserting
>> into
>> > mysql ?
>> >
>> > On Mon, Oct 12, 2015 at 1:54 PM, Susheel Kumar Gadalay
>> > <skgadalay@gmail.com>
>> > wrote:
>> >
>> >> It is better to have 2 bolts - mysql bolt and mongodb bolt.
>> >>
>> >> Let mysql bolt forward the tuple to mongodb bolt, so in case of error
>> >> it won't  emit.
>> >>
>> >> On 10/12/15, Ankur Garg <ankurgarg9@gmail.com> wrote:
>> >> > So I have a situation where the tuple received on Spout has to be
>> saved
>> >> to
>> >> > mysql database and mongoDb as well .
>> >> >
>> >> > What should be better . Using 1 bolt to save it into mysql and
>> MongoDb
>> >> or 2
>> >> > seperate Bolts (One for saving into mysql and other for saving into
>> >> Mongo).
>> >> >
>> >> > What happens when the exception occurs while saving into mysql ? I
>> >> believe
>> >> > I will get acknowledgement inside the fail method in my Spout . So
>> If I
>> >> > reprocess it using 2 bolts , I believe it will again be sent to Bolt
>> >> > for
>> >> > saving into Mongo database .
>> >> >
>> >> > If the above is true , will having 2 seperate bolts be of any
>> advantage
>> >> > ?
>> >> > how can I configure things so that Failure while inserting into mysql
>> >> does
>> >> > not impact inserting into MongoDb .
>> >> >
>> >> > Thanks
>> >> > Ankur
>> >> >
>> >> > On Sun, Oct 11, 2015 at 10:57 PM, Ravi Sharma <ping2ravi@gmail.com>
>> >> wrote:
>> >> >
>> >> >> That depends if ur spout error has affected jvm or normal
>> application
>> >> >> error
>> >> >>
>> >> >> performance issue in case of lot of errors, I don't think there
is
>> any
>> >> >> issue be coz of errors themselves but ofcourse if u r retrying
these
>> >> >> messages on failure then that means u will be processing lot of
>> >> >> messages
>> >> >> then normal and overall throughput will go down
>> >> >>
>> >> >> Ravi
>> >> >>
>> >> >> If ur topology has enabled acknowledgment that means spout will
>> always
>> >> >> receive
>> >> >> On 11 Oct 2015 18:15, "Ankur Garg" <ankurgarg9@gmail.com>
wrote:
>> >> >>
>> >> >>>
>> >> >>> Thanks for the reply Abhishek and Ravi .
>> >> >>>
>> >> >>> One question though , going with One topology with multiple
spouts
>> >> >>> ...What if something goes wrong in One spout or its associated
>> bolts
>> >> >>> ..
>> >> >>> Does it impact other Spout as well?
>> >> >>>
>> >> >>> Thanks
>> >> >>> Ankur
>> >> >>>
>> >> >>> On Sun, Oct 11, 2015 at 10:21 PM, Ravi Sharma <ping2ravi@gmail.com
>> >
>> >> >>> wrote:
>> >> >>>
>> >> >>>> No 100% right ansers , u will have to test and see what
will fit..
>> >> >>>>
>> >> >>>> persoanlly i wud suggest Multiple spouts in one Topology
and if
>> you
>> >> >>>> have
>> >> >>>> N node where topology will be running then each Spout(reading
from
>> >> >>>> one
>> >> >>>> queue) shud run N times in parallel.
>> >> >>>>
>> >> >>>> if 2 Queues and say 4 Nodes
>> >> >>>> then one topolgy
>> >> >>>> 4 Spouts reading from Queue1 in different nodes
>> >> >>>> 4 spouts reading from Queue2 in different nodes
>> >> >>>>
>> >> >>>> Ravi.
>> >> >>>>
>> >> >>>> On Sun, Oct 11, 2015 at 5:25 PM, Abhishek priya <
>> >> >>>> abhishek.priya@gmail.com> wrote:
>> >> >>>>
>> >> >>>>> I guess this is a question where there r no really
correct
>> answers.
>> >> >>>>> I'll certainly avoid#1 as it is better to keep logic
separate and
>> >> >>>>> lightweight.
>> >> >>>>>
>> >> >>>>> If your downstream bolts are same, then it makes senses
to keep
>> >> >>>>> them
>> >> >>>>> in
>> >> >>>>> same topology but if they r totally different, I'll
keep them in
>> >> >>>>> two
>> >> >>>>> different topologies. That will allow me to independently
deploy
>> >> >>>>> and
>> >> >>>>> scale
>> >> >>>>> the topology. But if the rest of logic is same I topology
scaling
>> >> >>>>> and
>> >> >>>>> resource utilization will be better with one topology.
>> >> >>>>>
>> >> >>>>> I hope this helps..
>> >> >>>>>
>> >> >>>>> Sent somehow....
>> >> >>>>>
>> >> >>>>> > On Oct 11, 2015, at 9:07 AM, Ankur Garg <ankurgarg9@gmail.com>
>> >> >>>>> > wrote:
>> >> >>>>> >
>> >> >>>>> > Hi ,
>> >> >>>>> >
>> >> >>>>> > So I have a situation where I want to read messages
from
>> >> >>>>> > different
>> >> >>>>> queues hosted in a Rabbitmq Server .
>> >> >>>>> >
>> >> >>>>> > Now , there are three ways which I can think to
leverage Apache
>> >> >>>>> > Storm
>> >> >>>>> here :-
>> >> >>>>> >
>> >> >>>>> > 1) Use the same Spout (say Spout A) to read messages
from
>> >> >>>>> > different
>> >> >>>>> queues and based on the messages received emit it to
different
>> >> >>>>> Bolts.
>> >> >>>>> >
>> >> >>>>> > 2) Use different Spout (Spout A and Spout B and
so on) within
>> the
>> >> >>>>> same topology (say Topology A) to read messages from
different
>> >> >>>>> queues
>> >> >>>>> .
>> >> >>>>> >
>> >> >>>>> > 3) Use Different Spouts one within eachTopology
(Topology A ,
>> >> >>>>> Topology B and so on) to read messages from different
queues .
>> >> >>>>> >
>> >> >>>>> > Which is the best way to process this considering
I want high
>> >> >>>>> throughput (more no of queue messages to be processed
>> concurrently)
>> >> >>>>> .
>> >> >>>>> >
>> >> >>>>> > Also , If In use same Topology for all Spouts
(currently though
>> >> >>>>> requirement is for 2 spouts)  will failure in one Spout
(or its
>> >> >>>>> associated
>> >> >>>>> Bolts) effect the second or will they both continue
working
>> >> separately
>> >> >>>>> even
>> >> >>>>> if some failure is in Spout B ?
>> >> >>>>> >
>> >> >>>>> > Cost wise , how much would it be to maintain two
different
>> >> >>>>> > topologies
>> >> >>>>> .
>> >> >>>>> >
>> >> >>>>> > Looking for inputs from members here.
>> >> >>>>> >
>> >> >>>>> > Thanks
>> >> >>>>> > Ankur
>> >> >>>>> >
>> >> >>>>> >
>> >> >>>>>
>> >> >>>>
>> >> >>>>
>> >> >>>
>> >> >
>> >>
>> >
>>
>
>

Mime
View raw message