incubator-s4-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Gowtham S <seldomgowt...@gmail.com>
Subject Re: Problem in running a same PE in multiple nodes
Date Sun, 24 Feb 2013 11:14:18 GMT
this is the input file for the above application..


On Sun, Feb 24, 2013 at 12:25 AM, Gowtham S <seldomgowtham@gmail.com> wrote:

> hi matthieu,
>
> i have no problem in app. and also 3 nodes are processing the whole data
> while one has stopped after some time..
>
> i have attached the sample project of our application. most of the
> description regarding the application is given in the attachment. if
> possible try checking it.. i ran it in a single machine with 4 nodes as
> said in previous message.
>
>
> i will describe what is actually happening ...
>
> i am having a record in which there are 18237 lines in which each line has
> 4 numbers. i m just trying to sort four numbers in each line based on the
> design of PE's as given in the document.
>
> same result one node stopped printing in the terminal while the other
> three continues it work.
>
> see this after reading the document..
>
> after seeing the document u will see the PE graph... in that Processing PE
> with Key ID "1" and Processing PE with Key ID "10" are running in the same
> node..
>
>  ProcessingPE with Key Id "10" should receive events from ProcessingPE of
> ID(0 and 1) but it mainly receives events from only PE with Key id "1" and
> not from Key Id  "0".
>
> so i cant get alternate events from both PE's. but i should receive events
> from both PE's one after the other as i  should sort 4 numbers in a single
> line. so PE in one node is not sending events that much quick to the PE in
> other nodes than to the PE in same node..
>
> so Processing PE with id "10" receives more events from PE with key ID "0"
> which is in the same node. so what i have to do in order to get events from
> both PE's with id "0" and "1" one after the other rather than majorly from
> "1" and then finally from "1"(i.e for e.g if "10" should receive 100 from
> both "0" and "1".. first 100 is received from "1" which is at the same node
> as "10" at that time only 2 or 3 received from "0". then the remaining from
> "0" is received. but i should receive 1 event from "0 " and 1 from "1" and
> so on ).
>
> "0" ,"1" - are Processing PE with Key ID "0" and "1".
>
>
>
>
>
>
>
> On Fri, Feb 22, 2013 at 7:03 PM, Matthieu Morel <mmorel@apache.org> wrote:
>
>> Thanks for the info,
>>
>> with the cluster status that you show - which look ok - is there any
>> problem with the app?
>>
>> Also, I would again recommend to check whether all data is processed as
>> expected, regardless of the location. Can you check that?
>>
>> Thanks,
>>
>> Matthieu
>>
>>
>> On Feb 22, 2013, at 14:02 , Gowtham S wrote:
>>
>> sorry for the wrong thread and gimp format..
>>
>> here i have attached the .png format of the screenshot which shows that
>> all the 4 nodes are active .
>>
>> and also i didnt modify the hash function. when ever i run it in 4 nodes,
>> it always map the 4 different keyed PE to four different nodes. i didnt
>> manually  change the hash function ..
>>
>>
>>
>> On Thu, Feb 21, 2013 at 1:30 AM, Matthieu Morel <mmorel@apache.org>wrote:
>>
>>> Hi Jihyoun,
>>>
>>> PE graphs with cycles can certainly get a bit tricky sometimes, but
>>> mainly because they can lead to deadlocks.
>>>
>>> I don't see how you'd lose events in the case you present if you don't
>>> lose or reconfigure nodes. In the very specific case of sending an event
>>> from PE instance a1 to the same PE instance a1, the event is simply put in
>>> an in-memory blocking queue.
>>>
>>> It would be great to understand in which conditions you can see this
>>> issue. Could you be more specific? (I can't reproduce that with simple
>>> tests, but I don't have your own settings/code/environment).
>>>
>>> Thanks,
>>>
>>> Matthieu
>>>
>>> On Feb 20, 2013, at 03:04 , JiHyoun Park wrote:
>>>
>>> Dear Matthieu
>>>
>>> I also experienced the same problem with Gowtham.
>>> Unlike him, I created a downstream to itself with only one key. (It was
>>> like a flag to be turned on/off.)
>>> I tested it and found sometimes one PE instance does not receive the
>>> event.
>>> I remember that it didn't depend on a specific node.
>>>
>>> At the time, I also wondered why.
>>> Since it's a very simple logic, you can test it by yourself without much
>>> efforts.
>>> It would be really appreciated if you can tell us the solution or at
>>> least the reason.
>>>
>>> Best Regards
>>> Jihyoun
>>>
>>>
>>>
>>> On Tue, Feb 19, 2013 at 11:38 PM, Matthieu Morel <mmorel@apache.org>wrote:
>>>
>>>> Hi,
>>>>
>>>> Can you provide more detail about a node not running correctly? what
>>>> does that mean to you, that it does not receive messages? Maybe that's
>>>> normal because of the distribution of keys. Or maybe you only have 3 active
>>>> nodes in the cluster. (you can check that with the s4 status tool)
>>>>
>>>> The distribution of keys across nodes is a simple hash+mod , which you
>>>> can override if needed.
>>>>
>>>> I would recommend to check whether all data is processed as expected,
>>>> regardless of the location.
>>>>
>>>> Let us know,
>>>>
>>>> thanks,
>>>>
>>>> Matthieu
>>>>
>>>>
>>>> On Feb 19, 2013, at 16:27 , Gowtham S wrote:
>>>>
>>>> Hi S4 community,
>>>>
>>>> I have been working with s4 piper for 3 months. now i am just trying to
>>>> make a downstream from a PE to the same PE with different Key Id.
>>>>
>>>> i just have a scenario in which i would have an unique PE whose
>>>> downstream is also to that PE  but each time the Key Id of the targeted PE
>>>> may vary. this continues until it satisfies a condition where it just stops.
>>>>
>>>> i started implementing this in a single machine with 4 nodes, as i had
>>>> 4 different keys for that unique PE(say 0,1,2,3). so now every node will
>>>> have that PE clone with one individual key Id from the available Key Ids.
>>>> one PE may send some events to PE in other  nodes , until certain condition
>>>> is reached where the process is stopped.
>>>>
>>>> now three nodes are running correctly except one node.. i wonder why ..
>>>> because all the nodes have same computational part to execute.
>>>>
>>>> please help me as how s4 deals while assigning a single PE to different
>>>> nodes with different key Id ..
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> On Tue, Feb 19, 2013 at 8:22 PM, Gowtham S <seldomgowtham@gmail.com>wrote:
>>>>
>>>>> Hi S4 community,
>>>>>
>>>>>
>>>>
>>>>
>>>
>>>
>> <s4image.png>
>>
>>
>>
>

Mime
View raw message