storm-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Rudraneel chakraborty <rudraneel.chakrabo...@gmail.com>
Subject Re: Multiple Spouts in Same topology or Topology per spout
Date Sun, 11 Oct 2015 19:57:09 GMT
Can you give me a situation where multiple dependent topology have been
used , say different topologies will infer a big complex event

On Sunday, 11 October 2015, Ravi Sharma <ping2ravi@gmail.com> wrote:

> That depends if ur spout error has affected jvm or normal application error
>
> performance issue in case of lot of errors, I don't think there is any
> issue be coz of errors themselves but ofcourse if u r retrying these
> messages on failure then that means u will be processing lot of messages
> then normal and overall throughput will go down
>
> Ravi
>
> If ur topology has enabled acknowledgment that means spout will always
> receive
> On 11 Oct 2015 18:15, "Ankur Garg" <ankurgarg9@gmail.com
> <javascript:_e(%7B%7D,'cvml','ankurgarg9@gmail.com');>> wrote:
>
>>
>> Thanks for the reply Abhishek and Ravi .
>>
>> One question though , going with One topology with multiple spouts
>> ...What if something goes wrong in One spout or its associated bolts ..
>> Does it impact other Spout as well?
>>
>> Thanks
>> Ankur
>>
>> On Sun, Oct 11, 2015 at 10:21 PM, Ravi Sharma <ping2ravi@gmail.com
>> <javascript:_e(%7B%7D,'cvml','ping2ravi@gmail.com');>> wrote:
>>
>>> No 100% right ansers , u will have to test and see what will fit..
>>>
>>> persoanlly i wud suggest Multiple spouts in one Topology and if you have
>>> N node where topology will be running then each Spout(reading from one
>>> queue) shud run N times in parallel.
>>>
>>> if 2 Queues and say 4 Nodes
>>> then one topolgy
>>> 4 Spouts reading from Queue1 in different nodes
>>> 4 spouts reading from Queue2 in different nodes
>>>
>>> Ravi.
>>>
>>> On Sun, Oct 11, 2015 at 5:25 PM, Abhishek priya <
>>> abhishek.priya@gmail.com
>>> <javascript:_e(%7B%7D,'cvml','abhishek.priya@gmail.com');>> wrote:
>>>
>>>> I guess this is a question where there r no really correct answers.
>>>> I'll certainly avoid#1 as it is better to keep logic separate and
>>>> lightweight.
>>>>
>>>> If your downstream bolts are same, then it makes senses to keep them in
>>>> same topology but if they r totally different, I'll keep them in two
>>>> different topologies. That will allow me to independently deploy and scale
>>>> the topology. But if the rest of logic is same I topology scaling and
>>>> resource utilization will be better with one topology.
>>>>
>>>> I hope this helps..
>>>>
>>>> Sent somehow....
>>>>
>>>> > On Oct 11, 2015, at 9:07 AM, Ankur Garg <ankurgarg9@gmail.com
>>>> <javascript:_e(%7B%7D,'cvml','ankurgarg9@gmail.com');>> wrote:
>>>> >
>>>> > Hi ,
>>>> >
>>>> > So I have a situation where I want to read messages from different
>>>> queues hosted in a Rabbitmq Server .
>>>> >
>>>> > Now , there are three ways which I can think to leverage Apache Storm
>>>> here :-
>>>> >
>>>> > 1) Use the same Spout (say Spout A) to read messages from different
>>>> queues and based on the messages received emit it to different Bolts.
>>>> >
>>>> > 2) Use different Spout (Spout A and Spout B and so on) within the
>>>> same topology (say Topology A) to read messages from different queues .
>>>> >
>>>> > 3) Use Different Spouts one within eachTopology (Topology A ,
>>>> Topology B and so on) to read messages from different queues .
>>>> >
>>>> > Which is the best way to process this considering I want high
>>>> throughput (more no of queue messages to be processed concurrently) .
>>>> >
>>>> > Also , If In use same Topology for all Spouts (currently though
>>>> requirement is for 2 spouts)  will failure in one Spout (or its associated
>>>> Bolts) effect the second or will they both continue working separately even
>>>> if some failure is in Spout B ?
>>>> >
>>>> > Cost wise , how much would it be to maintain two different topologies
>>>> .
>>>> >
>>>> > Looking for inputs from members here.
>>>> >
>>>> > Thanks
>>>> > Ankur
>>>> >
>>>> >
>>>>
>>>
>>>
>>

-- 
Rudraneel Chakraborty
Carleton University Real Time and Distributed Systems Reserach

Mime
View raw message