spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Denis Turdakov <turda...@ispras.ru>
Subject Bagel and partitioning
Date Fri, 16 Aug 2013 15:10:26 GMT
Hello everyone,

In ISPRAS (http://ispras.ru) we are working on several problems of 
social network analysis. In our work we are using Bagel implementation 
in Spark.

As one of ways to improve graph analysis performance we implement a 
graph partitioning algorithm based on balanced label propagation (BLP) 
algorithm developed in Facebook 
(http://dl.acm.org/citation.cfm?id=2433461). One problem is that it 
can't be directly integrated to Bagel since it does not know anything 
about graph edges. As a result interface for passing graph to the 
partitioner is different from interface for passing graph to Bagel. 
Because of this using partitioner requires more code modifications that 
it could.

So we thought about implementing another interface for processing graphs 
that would be aware of edges. Another reason for a different interface 
is that it may be very similar to GraphX so that switching between 
edge-partitioning and vertex-partitioning approaches in an application 
would be easier.


Could you please clarify following things for us:
1. Do you plan to continue development of Bagel?
2. Would you be interested in incorporating our graph processing 
interface into Spark if we implement it?
3. Is there any point in contributing BLP partitioner to Spark project 
in some way, e.g. as a Bagel partitioner?

Best regards,
Denis Turdakov


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message