spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Larry Xiao <xia...@sjtu.edu.cn>
Subject GraphX partitioning and threading details
Date Tue, 05 Aug 2014 06:57:09 GMT
Hi all,

about GraphX partitioning details and possible optimization.

  * Can you tell how are partitions distributed to nodes? And inside
    worker, how does partitions get allocated to threads?
      o Is it possible to make manual configuration, like partition A =>
        node 1, thread 1
  * How is memory organized among threads?
      o Can we exploit the shared memory to combine mirror cache on same
        node into one?
      o (our experiment shows that more partitions requires much more
        memory)

Larry


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message