spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Akhil Das <ak...@sigmoidanalytics.com>
Subject Re: "CANNOT FIND ADDRESS"
Date Sat, 01 Nov 2014 07:53:42 GMT
Tr this

spark.storage.memoryFraction    0.9
On 31 Oct 2014 20:27, "akhandeshi" <ami.khandeshi@gmail.com> wrote:

> Thanks for the pointers! I did tried but didn't seem to help...
>
> In my latest try, I am doing spark-submit local
>
> But see the same message in  spark App ui (4040)
> localhost       CANNOT FIND ADDRESS
>
> In the logs, I see a lot of in-memory map to disk.  I don't understand why
> that is the case.  There should be over 35 gb ram available for input that
> is not significantly large.  It may be link to the performance issues that
> I
> am seeing.  I have another post for seeing advice on  that.  It seems, I am
> not able to tune spark sufficiently to execute my process successfully.
>
> 14/10/31 13:45:12 INFO ExternalAppendOnlyMap: Thread 206 spilling in-memory
> map of 1777 MB to disk (2 times so
>  far)
> 14/10/31 13:45:12 INFO ExternalAppendOnlyMap: Thread 228 spilling in-memory
> map of 393 MB to disk (1 time so f
> ar)
> 14/10/31 13:45:12 INFO ExternalAppendOnlyMap: Thread 259 spilling in-memory
> map of 392 MB to disk (2 times so
> far)
> 14/10/31 13:45:14 INctorsBySecId();^ZFO ExternalAppendOnlyMap: Thread 230
> spilling in-memory map of 554 MB to
> disk (2 times so far)
> 14/10/31 13:45:15 INFO ExternalAppendOnlyMap: Thread 235 spilling in-memory
> map of 3990 MB to disk (1 time so
> far)
> 14/10/31 13:45:15 INFO ExternalAppendOnlyMap: Thread 236 spilling in-memory
> map of 2667 MB to disk (1 time so
> far)
> 14/10/31 13:45:17 INFO ExternalAppendOnlyMap: Thread 259 spilling in-memory
> map of 825 MB to disk (3 times so
> far)
> 14/10/31 13:45:24 INFO ExternalAppendOnlyMap: Thread 228 spilling in-memory
> map of 4618 MB to disk (2 times so
>  far)
> 14/10/31 13:45:26 INFO ExternalAppendOnlyMap: Thread 233 spilling in-memory
> map of 15869 MB to disk (1 time so
>  far)
> 14/10/31 13:45:37 INFO ExternalAppendOnlyMap: Thread 206 spilling in-memory
> map of 3026 MB to disk (3 times so
>  far)
> 14/10/31 13:45:48 INFO ExternalAppendOnlyMap: Thread 228 spilling in-memory
> map of 401 MB to disk (3 times so
> far)
> 14/10/31 13:45:48 INFO ExternalAppendOnlyMap: Thread 259 spilling in-memory
> map of 392 MB to disk (4 times so
>
> My spark properties are:
>
> Name    Value
> spark.akka.frameSize    50
> spark.akka.timeout      900
> spark.app.name  Simple File Merge Application
> spark.core.connection.ack.wait.timeout  900
> spark.default.parallelism       10
> spark.driver.host       spark-single.c.fi-mdd-poc.internal
> spark.driver.memory     35g
> spark.driver.port       40255
> spark.eventLog.enabled  true
> spark.fileserver.uri    http://10.240.106.135:59255
> spark.jars      file:/home/ami_khandeshi_gmail_com/SparkExample-1.0.jar
> spark.master    local[16]
> spark.scheduler.mode    FIFO
> spark.shuffle.consolidateFiles  true
> spark.storage.memoryFraction    0.3
> spark.tachyonStore.baseDir      /tmp
> spark.tachyonStore.folderName   spark-21ad0fd2-2177-48ce-9242-8dbb33f2a1f1
> spark.tachyonStore.url  tachyon://mdd:19998
>
>
>
> --
> View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/CANNOT-FIND-ADDRESS-tp17637p17824.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
>
>

Mime
View raw message