spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "lokesh.gidra" <lokesh.gi...@gmail.com>
Subject Re: "Spilling in-memory..." messages in log even with MEMORY_ONLY
Date Sun, 27 Jul 2014 17:29:14 GMT
I am comparing the total time spent in finishing the job. And What I am
comparing, to be precise, is on a 48-core machine. I am comparing the
performance of local[48] vs. standalone mode with 8 nodes of 6 cores each
(totalling 48 cores) on localhost. In this comparison, the standalone mode
outperforms local[48] substantially. When I did some troublshooting using
oprofile, I found that local[48] was spending much more time in writeObject0
as compared to standalone mode.

I am running the PageRank example provided in the package. 



--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Spilling-in-memory-messages-in-log-even-with-MEMORY-ONLY-tp10723p10743.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

Mime
View raw message