storm-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Susheel Kumar Gadalay <skgada...@gmail.com>
Subject Re: Multiple Spouts in Same topology or Topology per spout
Date Mon, 12 Oct 2015 09:12:06 GMT
Check and insert

On 10/12/15, Ankur Garg <ankurgarg9@gmail.com> wrote:
> But what if MongoDb bolt has some error , in that case I suppose the entire
> tuple will be replayed from Spout meaning it will have to redo the
> operation of inserting into sql . Is there a way I can skip inserting into
> mysql ?
>
> On Mon, Oct 12, 2015 at 1:54 PM, Susheel Kumar Gadalay
> <skgadalay@gmail.com>
> wrote:
>
>> It is better to have 2 bolts - mysql bolt and mongodb bolt.
>>
>> Let mysql bolt forward the tuple to mongodb bolt, so in case of error
>> it won't  emit.
>>
>> On 10/12/15, Ankur Garg <ankurgarg9@gmail.com> wrote:
>> > So I have a situation where the tuple received on Spout has to be saved
>> to
>> > mysql database and mongoDb as well .
>> >
>> > What should be better . Using 1 bolt to save it into mysql and MongoDb
>> or 2
>> > seperate Bolts (One for saving into mysql and other for saving into
>> Mongo).
>> >
>> > What happens when the exception occurs while saving into mysql ? I
>> believe
>> > I will get acknowledgement inside the fail method in my Spout . So If I
>> > reprocess it using 2 bolts , I believe it will again be sent to Bolt
>> > for
>> > saving into Mongo database .
>> >
>> > If the above is true , will having 2 seperate bolts be of any advantage
>> > ?
>> > how can I configure things so that Failure while inserting into mysql
>> does
>> > not impact inserting into MongoDb .
>> >
>> > Thanks
>> > Ankur
>> >
>> > On Sun, Oct 11, 2015 at 10:57 PM, Ravi Sharma <ping2ravi@gmail.com>
>> wrote:
>> >
>> >> That depends if ur spout error has affected jvm or normal application
>> >> error
>> >>
>> >> performance issue in case of lot of errors, I don't think there is any
>> >> issue be coz of errors themselves but ofcourse if u r retrying these
>> >> messages on failure then that means u will be processing lot of
>> >> messages
>> >> then normal and overall throughput will go down
>> >>
>> >> Ravi
>> >>
>> >> If ur topology has enabled acknowledgment that means spout will always
>> >> receive
>> >> On 11 Oct 2015 18:15, "Ankur Garg" <ankurgarg9@gmail.com> wrote:
>> >>
>> >>>
>> >>> Thanks for the reply Abhishek and Ravi .
>> >>>
>> >>> One question though , going with One topology with multiple spouts
>> >>> ...What if something goes wrong in One spout or its associated bolts
>> >>> ..
>> >>> Does it impact other Spout as well?
>> >>>
>> >>> Thanks
>> >>> Ankur
>> >>>
>> >>> On Sun, Oct 11, 2015 at 10:21 PM, Ravi Sharma <ping2ravi@gmail.com>
>> >>> wrote:
>> >>>
>> >>>> No 100% right ansers , u will have to test and see what will fit..
>> >>>>
>> >>>> persoanlly i wud suggest Multiple spouts in one Topology and if
you
>> >>>> have
>> >>>> N node where topology will be running then each Spout(reading from
>> >>>> one
>> >>>> queue) shud run N times in parallel.
>> >>>>
>> >>>> if 2 Queues and say 4 Nodes
>> >>>> then one topolgy
>> >>>> 4 Spouts reading from Queue1 in different nodes
>> >>>> 4 spouts reading from Queue2 in different nodes
>> >>>>
>> >>>> Ravi.
>> >>>>
>> >>>> On Sun, Oct 11, 2015 at 5:25 PM, Abhishek priya <
>> >>>> abhishek.priya@gmail.com> wrote:
>> >>>>
>> >>>>> I guess this is a question where there r no really correct answers.
>> >>>>> I'll certainly avoid#1 as it is better to keep logic separate
and
>> >>>>> lightweight.
>> >>>>>
>> >>>>> If your downstream bolts are same, then it makes senses to keep
>> >>>>> them
>> >>>>> in
>> >>>>> same topology but if they r totally different, I'll keep them
in
>> >>>>> two
>> >>>>> different topologies. That will allow me to independently deploy
>> >>>>> and
>> >>>>> scale
>> >>>>> the topology. But if the rest of logic is same I topology scaling
>> >>>>> and
>> >>>>> resource utilization will be better with one topology.
>> >>>>>
>> >>>>> I hope this helps..
>> >>>>>
>> >>>>> Sent somehow....
>> >>>>>
>> >>>>> > On Oct 11, 2015, at 9:07 AM, Ankur Garg <ankurgarg9@gmail.com>
>> >>>>> > wrote:
>> >>>>> >
>> >>>>> > Hi ,
>> >>>>> >
>> >>>>> > So I have a situation where I want to read messages from
>> >>>>> > different
>> >>>>> queues hosted in a Rabbitmq Server .
>> >>>>> >
>> >>>>> > Now , there are three ways which I can think to leverage
Apache
>> >>>>> > Storm
>> >>>>> here :-
>> >>>>> >
>> >>>>> > 1) Use the same Spout (say Spout A) to read messages from
>> >>>>> > different
>> >>>>> queues and based on the messages received emit it to different
>> >>>>> Bolts.
>> >>>>> >
>> >>>>> > 2) Use different Spout (Spout A and Spout B and so on)
within the
>> >>>>> same topology (say Topology A) to read messages from different
>> >>>>> queues
>> >>>>> .
>> >>>>> >
>> >>>>> > 3) Use Different Spouts one within eachTopology (Topology
A ,
>> >>>>> Topology B and so on) to read messages from different queues
.
>> >>>>> >
>> >>>>> > Which is the best way to process this considering I want
high
>> >>>>> throughput (more no of queue messages to be processed concurrently)
>> >>>>> .
>> >>>>> >
>> >>>>> > Also , If In use same Topology for all Spouts (currently
though
>> >>>>> requirement is for 2 spouts)  will failure in one Spout (or
its
>> >>>>> associated
>> >>>>> Bolts) effect the second or will they both continue working
>> separately
>> >>>>> even
>> >>>>> if some failure is in Spout B ?
>> >>>>> >
>> >>>>> > Cost wise , how much would it be to maintain two different
>> >>>>> > topologies
>> >>>>> .
>> >>>>> >
>> >>>>> > Looking for inputs from members here.
>> >>>>> >
>> >>>>> > Thanks
>> >>>>> > Ankur
>> >>>>> >
>> >>>>> >
>> >>>>>
>> >>>>
>> >>>>
>> >>>
>> >
>>
>

Mime
View raw message