spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Kaan Sancak <kaans...@gmail.com>
Subject Question on writing batch synchronized incremental graph algorithms
Date Tue, 14 Apr 2020 16:57:32 GMT
Hi all,
I have been trying to write batch-synchronized  incremental graph algorithms. More specifically,
I want to run an increment algorithm on a given data-set and when a new batch arrives, I want
to start the algorithm from last snapshot, and run the algorithm on the vertices that are
effected by the new batch. (Assuming that each batch contains a set of insertions/deletions
to the graph, effected vertices are the vertices whose neighbor set is effected by the insertions/deletions).

I found out a paper[1] which mentions about a framework called GraphTau built on top of GraphX,
and I also found out couple of Spark Summits[2] mention future code release. I have looked
but couldn’t find any code available to public. 
Is there anyone who can inform me about the subject? Or are there any features on GraphX currently
available for me to write such algorithms.

[1] Time-Evolving Graph Processing at Scale: https://www.researchgate.net/publication/305661018_Time-evolving_graph_processing_at_scale
<https://www.researchgate.net/publication/305661018_Time-evolving_graph_processing_at_scale>
.

[2] Spark Summit 2017: https://www.slideshare.net/SparkSummit/timeevolving-graph-processing-on-commodity-clusters-spark-summit-east-talk-by-anand-iyer
<https://www.slideshare.net/SparkSummit/timeevolving-graph-processing-on-commodity-clusters-spark-summit-east-talk-by-anand-iyer>

Best
Kaan
Mime
View raw message