spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From <michael.engl...@nomura.com>
Subject FW: No APPLICATION_COMPLETE file created in history server log location upon pyspark job success
Date Wed, 07 Jan 2015 11:38:44 GMT
Hi,

I am currently running pyspark jobs against Spark 1.1.0 on YARN. When I run example Java jobs
such as spark-pi, the following files get created:

bash-4.1$ tree spark-pi-1420624364958
spark-pi-1420624364958
âââ APPLICATION_COMPLETE
âââ EVENT_LOG_1
âââ SPARK_VERSION_1.1.0

0 directories, 3 files

However, when I run my pyspark job, no APPLICATION_COMPLETE file gets created.

bash-4.1$ tree pyspark-1420628130353
pyspark -1420628130353
âââ EVENT_LOG_1
âââ SPARK_VERSION_1.1.0

0 directories, 2 files

If I touch the file into this directory, it just appears as <not started> in the history
server UI.

I am submitting jobs using spark-submit for now:

bin/spark-submit --master yarn-client --executor-memory 4G --executor-cores 12 --num-executors
10 –queue highpriority <path to python file>


Is there a setting I am missing for this APPLICATION_COMPLETE file to be created when a pyspark
job completes?

Thanks,
Michael


This e-mail (including any attachments) is private and confidential, may contain proprietary
or privileged information and is intended for the named recipient(s) only. Unintended recipients
are strictly prohibited from taking action on the basis of information in this e-mail and
must contact the sender immediately, delete this e-mail (and all attachments) and destroy
any hard copies. Nomura will not accept responsibility or liability for the accuracy or completeness
of, or the presence of any virus or disabling code in, this e-mail. If verification is sought
please request a hard copy. Any reference to the terms of executed transactions should be
treated as preliminary only and subject to formal written confirmation by Nomura. Nomura reserves
the right to retain, monitor and intercept e-mail communications through its networks (subject
to and in accordance with applicable laws). No confidentiality or privilege is waived or lost
by Nomura by any mistransmission of this e-mail. Any reference to "Nomura" is a reference
to any entity in the Nomura Holdings, Inc. group. Please read our Electronic Communications
Legal Notice which forms part of this e-mail: http://www.Nomura.com/email_disclaimer.htm

Mime
View raw message