spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Yifan LI <iamyifa...@gmail.com>
Subject Re: How to measure communication between nodes in Spark Standalone Cluster?
Date Mon, 17 Nov 2014 15:32:12 GMT
I am not sure there is a direct way(an api in graphx, etc) to measure the number of transferred
vertex values among nodes during computation.

It might depend on:
- the operations in your application, e.g. only communicate with its immediate neighbours
for each vertex.
- the partition strategy you chose, wrt the vertices replication factor
- the distribution of partitions on cluster
...


Best,
Yifan LI
LIP6, UPMC, Paris





> On 17 Nov 2014, at 11:59, Hlib Mykhailenko <hlib.mykhailenko@inria.fr> wrote:
> 
> Hello,
> 
> I use Spark Standalone Cluster and I want to measure somehow internode communication.

> As I understood, Graphx transfers only vertices values. Am I right?  
> 
> But I do not want to get number of bytes which were transferred between any two nodes.
> So is  there way to measure how many values of vertices were transferred among nodes?
> 
> Thanks!
> 
> --
> Cordialement,
> Hlib Mykhailenko
> Doctorant à INRIA Sophia-Antipolis Méditerranée <http://www.inria.fr/centre/sophia/>
> 2004 Route des Lucioles BP93
> 06902 SOPHIA ANTIPOLIS cedex
> 


Mime
View raw message