spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Andrew Ash <and...@andrewash.com>
Subject Re: FW: No APPLICATION_COMPLETE file created in history server log location upon pyspark job success
Date Wed, 07 Jan 2015 15:25:54 GMT
Hi Michael,

I think you need to explicitly call sc.stop() on the spark context for it
to close down properly (this doesn't happen automatically).  See
https://issues.apache.org/jira/browse/SPARK-2972 for more details

Andrew

On Wed, Jan 7, 2015 at 3:38 AM, <michael.england@nomura.com> wrote:

>  Hi,
>
>
>
> I am currently running pyspark jobs against Spark 1.1.0 on YARN. When I
> run example Java jobs such as spark-pi, the following files get created:
>
>
>
> bash-4.1$ tree spark-pi-1420624364958
>
> spark-pi-1420624364958
>
> âââ APPLICATION_COMPLETE
>
> âââ EVENT_LOG_1
>
> âââ SPARK_VERSION_1.1.0
>
>
>
> 0 directories, 3 files
>
>
>
> However, when I run my pyspark job, no APPLICATION_COMPLETE file gets
> created.
>
>
>
> bash-4.1$ tree pyspark-1420628130353
>
> pyspark -1420628130353
>
> âââ EVENT_LOG_1
>
> âââ SPARK_VERSION_1.1.0
>
>
>
> 0 directories, 2 files
>
>
>
> If I touch the file into this directory, it just appears as <not started>
> in the history server UI.
>
>
>
> I am submitting jobs using spark-submit for now:
>
>
>
> *bin/spark-submit --master yarn-client --executor-memory 4G
> --executor-cores 12 --num-executors 10 –queue highpriority <path to python
> file>*
>
>
>
>
>
> Is there a setting I am missing for this APPLICATION_COMPLETE file to be
> created when a pyspark job completes?
>
>
>
> Thanks,
>
> Michael
>
> This e-mail (including any attachments) is private and confidential, may
> contain proprietary or privileged information and is intended for the named
> recipient(s) only. Unintended recipients are strictly prohibited from
> taking action on the basis of information in this e-mail and must contact
> the sender immediately, delete this e-mail (and all attachments) and
> destroy any hard copies. Nomura will not accept responsibility or liability
> for the accuracy or completeness of, or the presence of any virus or
> disabling code in, this e-mail. If verification is sought please request a
> hard copy. Any reference to the terms of executed transactions should be
> treated as preliminary only and subject to formal written confirmation by
> Nomura. Nomura reserves the right to retain, monitor and intercept e-mail
> communications through its networks (subject to and in accordance with
> applicable laws). No confidentiality or privilege is waived or lost by
> Nomura by any mistransmission of this e-mail. Any reference to "Nomura" is
> a reference to any entity in the Nomura Holdings, Inc. group. Please read
> our Electronic Communications Legal Notice which forms part of this e-mail:
> http://www.Nomura.com/email_disclaimer.htm
>

Mime
View raw message