[ https://issues.apache.org/jira/browse/SPARK-17707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15548785#comment-15548785 ] 

Maciej Bryński edited comment on SPARK-17707 at 10/5/16 1:58 PM:
-----------------------------------------------------------------

I did some tests.
{code}
mbrynski@jupyter:~$ cat submit.py
from pyspark.sql import SparkSession
spark = SparkSession.builder.enableHiveSupport().getOrCreate()

c = spark.table("some_table").count()

print(c)
{code}

1) Running without Spark Submit working with or without using WebUI 
{code}
$ echo $PYTHONPATH
/opt/spark/python:/opt/spark/python/lib/py4j-0.10.3-src.zip:
$ python3 submit.py
10000000

$ ps xf
27756 ?        S      0:00 sshd: mbrynski@pts/20
27757 pts/20   Ss     0:00  \_ -bash
12521 pts/20   Sl+    0:00      \_ python3 submit.py
12524 pts/20   Sl+    0:42          \_ /usr/lib/jvm/java-8-oracle//bin/java -cp /opt/spark/lib/mysql-connector-java-5.1.36-SNAPSHOT-bin.jar:/opt/spark/lib/dwh-hbase-connector.jar:/opt/spark/lib/hive-hbase-handler-1.2.1.spark.jar:/opt/spa
{code}

2) Running with Spark Submit and without Web UI
{code}
$ /opt/spark/bin/spark-submit submit.py
10000000

$ ps xf
27756 ?        S      0:00 sshd: mbrynski@pts/20
27757 pts/20   Ss     0:00  \_ -bash
14137 pts/20   Sl+    0:16      \_ /usr/lib/jvm/java-8-oracle//bin/java -cp /opt/spark/lib/mysql-connector-java-5.1.36-SNAPSHOT-bin.jar:/opt/spark/lib/dwh-hbase-connector.jar:/opt/spark/lib/hive-hbase-handler-1.2.1.spark.jar:/opt/spark/l
14257 pts/20   S+     0:00          \_ /usr/bin/python3 /home/mbrynski/submit.py
{code}

3) Running with Spark Submit and Web UI. Spark-submit stuck after job finished.
Processes after job's end. (compare to 2)
{code}
$ ps xf
27756 ?        S      0:00 sshd: mbrynski@pts/20
27757 pts/20   Ss     0:00  \_ -bash
16077 pts/20   Sl+    0:43      \_ /usr/lib/jvm/java-8-oracle//bin/java -cp /opt/spark/lib/mysql-connector-java-5.1.36-SNAPSHOT-bin.jar:/opt/spark/lib/dwh-hbase-connector.jar:/opt/spark/lib/hive-hbase-handler-1.2.1.spark.jar:/opt/spark/l
{code}


was (Author: maver1ck):
I did some tests.
{code}
mbrynski@jupyter:~$ cat submit.py
from pyspark.sql import SparkSession
spark = SparkSession.builder.enableHiveSupport().getOrCreate()

c = spark.table("some_table").count()

print(c)
{code}

1) Running without Spark Submit working with or without using WebUI 
{code}
$ echo $PYTHONPATH
/opt/spark/python:/opt/spark/python/lib/py4j-0.10.3-src.zip:
$ python3 submit.py
10000000

$ ps xf
27756 ?        S      0:00 sshd: mbrynski@pts/20
27757 pts/20   Ss     0:00  \_ -bash
12521 pts/20   Sl+    0:00      \_ python3 submit.py
12524 pts/20   Sl+    0:42          \_ /usr/lib/jvm/java-8-oracle//bin/java -cp /opt/spark/lib/mysql-connector-java-5.1.36-SNAPSHOT-bin.jar:/opt/spark/lib/dwh-hbase-connector.jar:/opt/spark/lib/hive-hbase-handler-1.2.1.spark.jar:/opt/spa
{code}

2) Running with Spark Submit and without Web UI
{code}
$ /opt/spark/bin/spark-submit submit.py
10000000

$ ps xf
27756 ?        S      0:00 sshd: mbrynski@pts/20
27757 pts/20   Ss     0:00  \_ -bash
14137 pts/20   Sl+    0:16      \_ /usr/lib/jvm/java-8-oracle//bin/java -cp /opt/spark/lib/mysql-connector-java-5.1.36-SNAPSHOT-bin.jar:/opt/spark/lib/dwh-hbase-connector.jar:/opt/spark/lib/hive-hbase-handler-1.2.1.spark.jar:/opt/spark/l
14257 pts/20   S+     0:00          \_ /usr/bin/python3 /home/mbrynski/submit.py
{code}

3) Running with Spark Submit and Web UI. Processes after job's end. (compare to 2)
{code}
$ ps xf
27756 ?        S      0:00 sshd: mbrynski@pts/20
27757 pts/20   Ss     0:00  \_ -bash
16077 pts/20   Sl+    0:43      \_ /usr/lib/jvm/java-8-oracle//bin/java -cp /opt/spark/lib/mysql-connector-java-5.1.36-SNAPSHOT-bin.jar:/opt/spark/lib/dwh-hbase-connector.jar:/opt/spark/lib/hive-hbase-handler-1.2.1.spark.jar:/opt/spark/l
{code}

> Web UI prevents spark-submit application to be finished
> -------------------------------------------------------
>
>                 Key: SPARK-17707
>                 URL: https://issues.apache.org/jira/browse/SPARK-17707
>             Project: Spark
>          Issue Type: Bug
>    Affects Versions: 2.0.0, 2.0.1
>            Reporter: Nick Orka
>
> Here are re-production steps:
> 1. create any scala spark application which will work long enough to open the application details in Web UI
> 2. run spark-submit command for standalone cluster, like: --master spark:\\localhost:7077
> 3. open running application details in Web UI, like: localhost:4040
> 4. spark-submit will never finish, you will have to kill the process
> Cause: The application creates a thread with infinite loop for web UI communication  and never stops it. The application is waiting for the thread to be finished instead, even if you close the web page.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org