storm-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Susheel Kumar Gadalay <skgada...@gmail.com>
Subject Re: Multiple Spouts in Same topology or Topology per spout
Date Mon, 12 Oct 2015 08:24:52 GMT
It is better to have 2 bolts - mysql bolt and mongodb bolt.

Let mysql bolt forward the tuple to mongodb bolt, so in case of error
it won't  emit.

On 10/12/15, Ankur Garg <ankurgarg9@gmail.com> wrote:
> So I have a situation where the tuple received on Spout has to be saved to
> mysql database and mongoDb as well .
>
> What should be better . Using 1 bolt to save it into mysql and MongoDb or 2
> seperate Bolts (One for saving into mysql and other for saving into Mongo).
>
> What happens when the exception occurs while saving into mysql ? I believe
> I will get acknowledgement inside the fail method in my Spout . So If I
> reprocess it using 2 bolts , I believe it will again be sent to Bolt for
> saving into Mongo database .
>
> If the above is true , will having 2 seperate bolts be of any advantage ?
> how can I configure things so that Failure while inserting into mysql does
> not impact inserting into MongoDb .
>
> Thanks
> Ankur
>
> On Sun, Oct 11, 2015 at 10:57 PM, Ravi Sharma <ping2ravi@gmail.com> wrote:
>
>> That depends if ur spout error has affected jvm or normal application
>> error
>>
>> performance issue in case of lot of errors, I don't think there is any
>> issue be coz of errors themselves but ofcourse if u r retrying these
>> messages on failure then that means u will be processing lot of messages
>> then normal and overall throughput will go down
>>
>> Ravi
>>
>> If ur topology has enabled acknowledgment that means spout will always
>> receive
>> On 11 Oct 2015 18:15, "Ankur Garg" <ankurgarg9@gmail.com> wrote:
>>
>>>
>>> Thanks for the reply Abhishek and Ravi .
>>>
>>> One question though , going with One topology with multiple spouts
>>> ...What if something goes wrong in One spout or its associated bolts ..
>>> Does it impact other Spout as well?
>>>
>>> Thanks
>>> Ankur
>>>
>>> On Sun, Oct 11, 2015 at 10:21 PM, Ravi Sharma <ping2ravi@gmail.com>
>>> wrote:
>>>
>>>> No 100% right ansers , u will have to test and see what will fit..
>>>>
>>>> persoanlly i wud suggest Multiple spouts in one Topology and if you
>>>> have
>>>> N node where topology will be running then each Spout(reading from one
>>>> queue) shud run N times in parallel.
>>>>
>>>> if 2 Queues and say 4 Nodes
>>>> then one topolgy
>>>> 4 Spouts reading from Queue1 in different nodes
>>>> 4 spouts reading from Queue2 in different nodes
>>>>
>>>> Ravi.
>>>>
>>>> On Sun, Oct 11, 2015 at 5:25 PM, Abhishek priya <
>>>> abhishek.priya@gmail.com> wrote:
>>>>
>>>>> I guess this is a question where there r no really correct answers.
>>>>> I'll certainly avoid#1 as it is better to keep logic separate and
>>>>> lightweight.
>>>>>
>>>>> If your downstream bolts are same, then it makes senses to keep them
>>>>> in
>>>>> same topology but if they r totally different, I'll keep them in two
>>>>> different topologies. That will allow me to independently deploy and
>>>>> scale
>>>>> the topology. But if the rest of logic is same I topology scaling and
>>>>> resource utilization will be better with one topology.
>>>>>
>>>>> I hope this helps..
>>>>>
>>>>> Sent somehow....
>>>>>
>>>>> > On Oct 11, 2015, at 9:07 AM, Ankur Garg <ankurgarg9@gmail.com>
>>>>> > wrote:
>>>>> >
>>>>> > Hi ,
>>>>> >
>>>>> > So I have a situation where I want to read messages from different
>>>>> queues hosted in a Rabbitmq Server .
>>>>> >
>>>>> > Now , there are three ways which I can think to leverage Apache
>>>>> > Storm
>>>>> here :-
>>>>> >
>>>>> > 1) Use the same Spout (say Spout A) to read messages from different
>>>>> queues and based on the messages received emit it to different Bolts.
>>>>> >
>>>>> > 2) Use different Spout (Spout A and Spout B and so on) within the
>>>>> same topology (say Topology A) to read messages from different queues
>>>>> .
>>>>> >
>>>>> > 3) Use Different Spouts one within eachTopology (Topology A ,
>>>>> Topology B and so on) to read messages from different queues .
>>>>> >
>>>>> > Which is the best way to process this considering I want high
>>>>> throughput (more no of queue messages to be processed concurrently) .
>>>>> >
>>>>> > Also , If In use same Topology for all Spouts (currently though
>>>>> requirement is for 2 spouts)  will failure in one Spout (or its
>>>>> associated
>>>>> Bolts) effect the second or will they both continue working separately
>>>>> even
>>>>> if some failure is in Spout B ?
>>>>> >
>>>>> > Cost wise , how much would it be to maintain two different
>>>>> > topologies
>>>>> .
>>>>> >
>>>>> > Looking for inputs from members here.
>>>>> >
>>>>> > Thanks
>>>>> > Ankur
>>>>> >
>>>>> >
>>>>>
>>>>
>>>>
>>>
>

Mime
View raw message