spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Kürşat Kurt <kur...@kursatkurt.com>
Subject RE: Out of memory at 60GB free memory.
Date Mon, 07 Nov 2016 18:37:49 GMT
Spark is not running  on mesos, runing only client mode.

 

From: Rodrick Brown [mailto:rodrick@orchardplatform.com] 
Sent: Monday, November 7, 2016 8:15 PM
To: Kürşat Kurt <kursat@kursatkurt.com>
Cc: Sean Owen <sowen@cloudera.com>; User <user@spark.apache.org>
Subject: Re: Out of memory at 60GB free memory.

 

You should also set memory overhead i.e. --conf spark.mesos.executor.memoryOverhead=${EXECUTOR_MEM}
* .10

 

On Mon, Nov 7, 2016 at 6:51 AM, Kürşat Kurt <kursat@kursatkurt.com <mailto:kursat@kursatkurt.com>
> wrote:

I understand that i shoud set the executor memory. I tried with the parameters below but OOM
still occures...

./spark-submit --class main.scala.Test1 --master local[8]  --driver-memory 20g --executor-memory
20g

 

From: Sean Owen [mailto:sowen@cloudera.com <mailto:sowen@cloudera.com> ] 
Sent: Monday, November 7, 2016 12:21 PM
To: Kürşat Kurt <kursat@kursatkurt.com <mailto:kursat@kursatkurt.com> >; user@spark.apache.org
<mailto:user@spark.apache.org> 
Subject: Re: Out of memory at 60GB free memory.

 

You say "out of memory", and you allocate a huge amount of driver memory, but, it's your executor
that's running out of memory. You want --executor-memory. You can't set it after the driver
has run.

On Mon, Nov 7, 2016 at 5:35 AM Kürşat Kurt <kursat@kursatkurt.com <mailto:kursat@kursatkurt.com>
> wrote:

Hi;

I am trying to use Naive Bayes for multi-class classification.

I am getting OOM at “pipeline.fit(train)” line. When i submit the code, everything is
ok so far the stage “collect at NaiveBayes.scala:400”.

At this stage, starting 375 tasks very fast and going slowing down at this point. Task count
could not became 500, getting OOM at 380-390th task.

 





 

-- 

 <http://www.orchardplatform.com/> 

Rodrick Brown / DevOPs

9174456839 / rodrick@orchardplatform.com <mailto:rodrick@orchardplatform.com> 

Orchard Platform 
101 5th Avenue, 4th Floor, New York, NY

 

NOTICE TO RECIPIENTS: This communication is confidential and intended for the use of the addressee
only. If you are not an intended recipient of this communication, please delete it immediately
and notify the sender by return email. Unauthorized reading, dissemination, distribution or
copying of this communication is prohibited. This communication does not constitute an offer
to sell or a solicitation of an indication of interest to purchase any loan, security or any
other financial product or instrument, nor is it an offer to sell or a solicitation of an
indication of interest to purchase any products or services to any persons who are prohibited
from receiving such information under applicable law. The contents of this communication may
not be accurate or complete and are subject to change without notice. As such, Orchard App,
Inc. (including its subsidiaries and affiliates, "Orchard") makes no representation regarding
the accuracy or completeness of the information contained herein. The intended recipient is
advised to consult its own professional advisors, including those specializing in legal, tax
and accounting matters. Orchard does not provide legal, tax or accounting advice.


Mime
View raw message