Tr this

spark.storage.memoryFraction    0.9

On 31 Oct 2014 20:27, "akhandeshi" <ami.khandeshi@gmail.com> wrote:
Thanks for the pointers! I did tried but didn't seem to help...

In my latest try, I am doing spark-submit local

But see the same message in  spark App ui (4040)
localhost       CANNOT FIND ADDRESS

In the logs, I see a lot of in-memory map to disk.  I don't understand why
that is the case.  There should be over 35 gb ram available for input that
is not significantly large.  It may be link to the performance issues that I
am seeing.  I have another post for seeing advice on  that.  It seems, I am
not able to tune spark sufficiently to execute my process successfully.

14/10/31 13:45:12 INFO ExternalAppendOnlyMap: Thread 206 spilling in-memory
map of 1777 MB to disk (2 times so
 far)
14/10/31 13:45:12 INFO ExternalAppendOnlyMap: Thread 228 spilling in-memory
map of 393 MB to disk (1 time so f
ar)
14/10/31 13:45:12 INFO ExternalAppendOnlyMap: Thread 259 spilling in-memory
map of 392 MB to disk (2 times so
far)
14/10/31 13:45:14 INctorsBySecId();^ZFO ExternalAppendOnlyMap: Thread 230
spilling in-memory map of 554 MB to
disk (2 times so far)
14/10/31 13:45:15 INFO ExternalAppendOnlyMap: Thread 235 spilling in-memory
map of 3990 MB to disk (1 time so
far)
14/10/31 13:45:15 INFO ExternalAppendOnlyMap: Thread 236 spilling in-memory
map of 2667 MB to disk (1 time so
far)
14/10/31 13:45:17 INFO ExternalAppendOnlyMap: Thread 259 spilling in-memory
map of 825 MB to disk (3 times so
far)
14/10/31 13:45:24 INFO ExternalAppendOnlyMap: Thread 228 spilling in-memory
map of 4618 MB to disk (2 times so
 far)
14/10/31 13:45:26 INFO ExternalAppendOnlyMap: Thread 233 spilling in-memory
map of 15869 MB to disk (1 time so
 far)
14/10/31 13:45:37 INFO ExternalAppendOnlyMap: Thread 206 spilling in-memory
map of 3026 MB to disk (3 times so
 far)
14/10/31 13:45:48 INFO ExternalAppendOnlyMap: Thread 228 spilling in-memory
map of 401 MB to disk (3 times so
far)
14/10/31 13:45:48 INFO ExternalAppendOnlyMap: Thread 259 spilling in-memory
map of 392 MB to disk (4 times so

My spark properties are:

Name    Value
spark.akka.frameSize    50
spark.akka.timeout      900
spark.app.name  Simple File Merge Application
spark.core.connection.ack.wait.timeout  900
spark.default.parallelism       10
spark.driver.host       spark-single.c.fi-mdd-poc.internal
spark.driver.memory     35g
spark.driver.port       40255
spark.eventLog.enabled  true
spark.fileserver.uri    http://10.240.106.135:59255
spark.jars      file:/home/ami_khandeshi_gmail_com/SparkExample-1.0.jar
spark.master    local[16]
spark.scheduler.mode    FIFO
spark.shuffle.consolidateFiles  true
spark.storage.memoryFraction    0.3
spark.tachyonStore.baseDir      /tmp
spark.tachyonStore.folderName   spark-21ad0fd2-2177-48ce-9242-8dbb33f2a1f1
spark.tachyonStore.url  tachyon://mdd:19998



--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/CANNOT-FIND-ADDRESS-tp17637p17824.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org