[ https://issues.apache.org/jira/browse/SPARK-17707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15548785#comment-15548785
]
Maciej Bryński commented on SPARK-17707:
----------------------------------------
I did some tests.
{code}
mbrynski@jupyter:~$ cat submit.py
from pyspark.sql import SparkSession
spark = SparkSession.builder.enableHiveSupport().getOrCreate()
c = spark.table("some_table").count()
print(c)
{code}
1) Running without Spark Submit working with or without using WebUI
{code}
$ echo $PYTHONPATH
/opt/spark/python:/opt/spark/python/lib/py4j-0.10.3-src.zip:
$ python3 submit.py
10000000
$ ps xf
27756 ? S 0:00 sshd: mbrynski@pts/20
27757 pts/20 Ss 0:00 \_ -bash
12521 pts/20 Sl+ 0:00 \_ python3 submit.py
12524 pts/20 Sl+ 0:42 \_ /usr/lib/jvm/java-8-oracle//bin/java -cp /opt/spark/lib/mysql-connector-java-5.1.36-SNAPSHOT-bin.jar:/opt/spark/lib/dwh-hbase-connector.jar:/opt/spark/lib/hive-hbase-handler-1.2.1.spark.jar:/opt/spa
{code}
2) Running with Spark Submit and without Web UI
{code}
$ /opt/spark/bin/spark-submit submit.py
10000000
$ ps xf
27756 ? S 0:00 sshd: mbrynski@pts/20
27757 pts/20 Ss 0:00 \_ -bash
14137 pts/20 Sl+ 0:16 \_ /usr/lib/jvm/java-8-oracle//bin/java -cp /opt/spark/lib/mysql-connector-java-5.1.36-SNAPSHOT-bin.jar:/opt/spark/lib/dwh-hbase-connector.jar:/opt/spark/lib/hive-hbase-handler-1.2.1.spark.jar:/opt/spark/l
14257 pts/20 S+ 0:00 \_ /usr/bin/python3 /home/mbrynski/submit.py
{code}
3) Running with Spark Submit and Web UI. Processes after job's end. (compare to 2)
{code}
$ ps xf
27756 ? S 0:00 sshd: mbrynski@pts/20
27757 pts/20 Ss 0:00 \_ -bash
16077 pts/20 Sl+ 0:43 \_ /usr/lib/jvm/java-8-oracle//bin/java -cp /opt/spark/lib/mysql-connector-java-5.1.36-SNAPSHOT-bin.jar:/opt/spark/lib/dwh-hbase-connector.jar:/opt/spark/lib/hive-hbase-handler-1.2.1.spark.jar:/opt/spark/l
{code}
> Web UI prevents spark-submit application to be finished
> -------------------------------------------------------
>
> Key: SPARK-17707
> URL: https://issues.apache.org/jira/browse/SPARK-17707
> Project: Spark
> Issue Type: Bug
> Affects Versions: 2.0.0, 2.0.1
> Reporter: Nick Orka
>
> Here are re-production steps:
> 1. create any scala spark application which will work long enough to open the application
details in Web UI
> 2. run spark-submit command for standalone cluster, like: --master spark:\\localhost:7077
> 3. open running application details in Web UI, like: localhost:4040
> 4. spark-submit will never finish, you will have to kill the process
> Cause: The application creates a thread with infinite loop for web UI communication
and never stops it. The application is waiting for the thread to be finished instead, even
if you close the web page.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org
|