spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jacek Laskowski <ja...@japila.pl>
Subject Re: Container preempted by scheduler - Spark job error
Date Thu, 02 Jun 2016 10:34:44 GMT
Hi,

Few things for closer examination:

* is yarn master URL accepted in 1.3? I thought it was only in later
releases. Since you're seeing the issue it seems it does work.

* I've never seen specifying confs using a single string. Can you check in
the Web ui they're applied?

* what about this <custom> in class?

Can you publish the application logs from the driver and executors?

Jacek
On 2 Jun 2016 10:33 a.m., "Prabeesh K." <prabsmails@gmail.com> wrote:

> Hi I am using the below command to run a spark job and I get an error like
> "Container preempted by scheduler"
>
> I am not sure if it's related to the wrong usage of Memory:
>
> nohup ~/spark1.3/bin/spark-submit \ --num-executors 50 \ --master yarn \
> --deploy-mode cluster \ --queue adhoc \ --driver-memory 18G \
> --executor-memory 12G \ --class main.ru.<custom>.bigdata.externalchurn.Main
> \ --conf "spark.task.maxFailures=100" \ --conf
> "spark.yarn.max.executor.failures=10000" \ --conf "spark.executor.cores=1"
> \ --conf "spark.akka.frameSize=50" \ --conf
> "spark.storage.memoryFraction=0.5" \ --conf
> "spark.driver.maxResultSize=10G" \
> ~/external-flow/externalChurn-1.0-SNAPSHOT-shaded.jar \
> prepareTraining=true \ prepareTrainingMNP=true \ prepareMap=false \
> bouldozerMode=true \ &> ~/external-flow/run.log & echo "STARTED" tail -f
> ~/external-flow/run.log
>
> Thanks,
>
>
>
>
>

Mime
View raw message