incubator-s4-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Dingyu Yang <yangdin...@gmail.com>
Subject Re: Fault tolerance and communication
Date Thu, 21 Mar 2013 10:56:49 GMT
Hi,Matthieu
The version is S4 0.6 RC2.
As I know, if the key is null, every event is send to one node, depending
on round robin.
 I just use the code to send the event to app cluster:
getRemoteStream().put(event);
The adapter is not keyed. The splitPE is singleton as twitter example.
What do you mean : a given operation is repeated for each target node.
It doesn't make sense of increasing node, the time increases.

Dingyu

>
> > Another problem is that the communication between adapter and app.
> > I test the experiment of word count, a 500M file with 80775764 words.
> > multiple nodes for app partitions, one node for adapter.
> > I test one adatper node and one app node, the adapter sending all the
> words
> > is done with 35 seconds.
> > one adatper node and two app node, the adapter is done with 61 seconds.
> > one adatper node and three app node, the adapter is done with 95 seconds.
> >
> > The adapter node is a same node and same program.
> > The time of adapter should be same or less with increasing app nodes,
> since
> > its processing ability has increased.
> > I don't know what the problem is.
>
> There were some extra copies in S4 0.5 so if you are using that version it
> could be an explanation.
>
> The pattern is quite clear though (linear increase with number of nodes)
> and it should be easy to spot the issue. Looks like a given operation is
> repeated for each target node. Are you broadcasting to all nodes? Are the
> events from the adapter keyed? Is there something specifically related in
> your adapter app code or adapter app graph?
>
>
> Regards,
>
> Matthieu
>
>

Mime
View raw message