<div class="socmaildefaultfont" dir="ltr" style="font-family:Arial, Helvetica, sans-serif;font-size:10.5pt"
><div dir="ltr" ><div dir="ltr" style="outline: none; color: rgb(18, 18, 18);
font-family: Arial, Helvetica, sans-serif; font-size: 14px; font-style: normal; font-variant-ligatures:
normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align:
start; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing:
0px; -webkit-text-stroke-width: 0px; background-color: rgb(255, 255, 255); text-decoration-style:
initial; text-decoration-color: initial;" > </div>
<div dir="ltr" style="outline: none; color: rgb(18, 18, 18); font-family: Arial, Helvetica,
sans-serif; font-size: 14px; font-style: normal; font-variant-ligatures: normal; font-variant-caps:
normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent:
0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-stroke-width:
0px; background-color: rgb(255, 255, 255); text-decoration-style: initial; text-decoration-color:
initial;" > </div>
<div dir="ltr" style="outline: none; color: rgb(18, 18, 18); font-family: Arial, Helvetica,
sans-serif; font-size: 14px; font-style: normal; font-variant-ligatures: normal; font-variant-caps:
normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent:
0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-stroke-width:
0px; background-color: rgb(255, 255, 255); text-decoration-style: initial; text-decoration-color:
initial;" >I have python jupyter notebook setup to create a
spark context by default, and<span> </span><strong style="outline:
none;" >sometimes</strong><span> </span>these fail with the
following error:</div>
<div dir="ltr" style="outline: none; color: rgb(18, 18, 18); font-family: Arial, Helvetica,
sans-serif; font-size: 14px; font-style: normal; font-variant-ligatures: normal; font-variant-caps:
normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent:
0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-stroke-width:
0px; background-color: rgb(255, 255, 255); text-decoration-style: initial; text-decoration-color:
initial;" > </div>
<div dir="ltr" style="outline: none; color: rgb(18, 18, 18); font-family: Arial, Helvetica,
sans-serif; font-size: 14px; font-style: normal; font-variant-ligatures: normal; font-variant-caps:
normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent:
0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-stroke-width:
0px; background-color: rgb(255, 255, 255); text-decoration-style: initial; text-decoration-color:
initial;" ><div style="outline: none;" ><font face="Default Monospace,Courier
New,Courier,monospace" size="2" style="outline: none;" ><code style="outline: none;
box-sizing: border-box; font-family: monospace, monospace; font-size: 1em;" >18/04/30 18:03:27
WARN Utils: Service 'sparkDriver' could not bind on port 0. Attempting port 1.<br style="outline:
none;" >18/04/30 18:03:27 ERROR SparkContext: Error initializing SparkContext.<br style="outline:
none;" >java.net.BindException: Cannot assign requested address: Service 'sparkDriver'
failed after 100 retries! Consider explicitly setting the appropriate port for the service
'sparkDriver' (for example spark.ui.port for SparkUI) to an available port or increasing spark.port.maxRetries.</code></font></div></div>
<div dir="ltr" style="outline: none; color: rgb(18, 18, 18); font-family: Arial, Helvetica,
sans-serif; font-size: 14px; font-style: normal; font-variant-ligatures: normal; font-variant-caps:
normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent:
0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-stroke-width:
0px; background-color: rgb(255, 255, 255); text-decoration-style: initial; text-decoration-color:
initial;" >I have tracked it down to two possible settings that may cause this in spark
2.0.2, client mode, standalone cluster setup, running in kubernetes:</div>
<div dir="ltr" style="outline: none; color: rgb(18, 18, 18); font-family: Arial, Helvetica,
sans-serif; font-size: 14px; font-style: normal; font-variant-ligatures: normal; font-variant-caps:
normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent:
0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-stroke-width:
0px; background-color: rgb(255, 255, 255); text-decoration-style: initial; text-decoration-color:
initial;" > </div>
<div dir="ltr" style="outline: none; color: rgb(18, 18, 18); font-family: Arial, Helvetica,
sans-serif; font-size: 14px; font-style: normal; font-variant-ligatures: normal; font-variant-caps:
normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent:
0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-stroke-width:
0px; background-color: rgb(255, 255, 255); text-decoration-style: initial; text-decoration-color:
initial;" >spark.driver.port - we don't set it, so it should be random</div>
<div dir="ltr" style="outline: none; color: rgb(18, 18, 18); font-family: Arial, Helvetica,
sans-serif; font-size: 14px; font-style: normal; font-variant-ligatures: normal; font-variant-caps:
normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent:
0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-stroke-width:
0px; background-color: rgb(255, 255, 255); text-decoration-style: initial; text-decoration-color:
initial;" >spark.ui.port - we set spark.ui.enabled=false so it should not try to bind to
this port.</div>
<div dir="ltr" style="outline: none; color: rgb(18, 18, 18); font-family: Arial, Helvetica,
sans-serif; font-size: 14px; font-style: normal; font-variant-ligatures: normal; font-variant-caps:
normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent:
0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-stroke-width:
0px; background-color: rgb(255, 255, 255); text-decoration-style: initial; text-decoration-color:
initial;" > </div>
<div dir="ltr" style="outline: none; color: rgb(18, 18, 18); font-family: Arial, Helvetica,
sans-serif; font-size: 14px; font-style: normal; font-variant-ligatures: normal; font-variant-caps:
normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent:
0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-stroke-width:
0px; background-color: rgb(255, 255, 255); text-decoration-style: initial; text-decoration-color:
initial;" >Short story is I do not know which one spark gets confused about, and looking
at spark code not clear how spark.ui.port would cause this even if the error message
lists it as a possible cause.</div>
<div dir="ltr" style="outline: none; color: rgb(18, 18, 18); font-family: Arial, Helvetica,
sans-serif; font-size: 14px; font-style: normal; font-variant-ligatures: normal; font-variant-caps:
normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent:
0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-stroke-width:
0px; background-color: rgb(255, 255, 255); text-decoration-style: initial; text-decoration-color:
initial;" > </div>
<div dir="ltr" style="outline: none; color: rgb(18, 18, 18); font-family: Arial, Helvetica,
sans-serif; font-size: 14px; font-style: normal; font-variant-ligatures: normal; font-variant-caps:
normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent:
0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-stroke-width:
0px; background-color: rgb(255, 255, 255); text-decoration-style: initial; text-decoration-color:
initial;" ><strong style="outline: none;" >Question 1:</strong><span> </span>have
you seen this before?</div>
<div dir="ltr" style="outline: none; color: rgb(18, 18, 18); font-family: Arial, Helvetica,
sans-serif; font-size: 14px; font-style: normal; font-variant-ligatures: normal; font-variant-caps:
normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent:
0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-stroke-width:
0px; background-color: rgb(255, 255, 255); text-decoration-style: initial; text-decoration-color:
initial;" ><strong style="outline: none;" >Question 2:</strong><span> </span>how
do I trace the spark driver process? It seems that I can only set the sc.logLevel
after the spark context is created, but I need to trace before the spark context is created.</div>
<div dir="ltr" style="outline: none; color: rgb(18, 18, 18); font-family: Arial, Helvetica,
sans-serif; font-size: 14px; font-style: normal; font-variant-ligatures: normal; font-variant-caps:
normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent:
0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-stroke-width:
0px; background-color: rgb(255, 255, 255); text-decoration-style: initial; text-decoration-color:
initial;" > </div>
<div dir="ltr" style="outline: none; color: rgb(18, 18, 18); font-family: Arial, Helvetica,
sans-serif; font-size: 14px; font-style: normal; font-variant-ligatures: normal; font-variant-caps:
normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent:
0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-stroke-width:
0px; background-color: rgb(255, 255, 255); text-decoration-style: initial; text-decoration-color:
initial;" >I created a log4j.properties file in the spark/conf directory and set it to
TRACE but that only gets picked up when I run a Scala jupyter notebook, not when
I run a python juypyter notebook, and I haven't been able to find out how to turn the same
level of tracing for a spark-driver process started via a python jupyter notebook.</div>
<div dir="ltr" style="outline: none; color: rgb(18, 18, 18); font-family: Arial, Helvetica,
sans-serif; font-size: 14px; font-style: normal; font-variant-ligatures: normal; font-variant-caps:
normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent:
0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-stroke-width:
0px; background-color: rgb(255, 255, 255); text-decoration-style: initial; text-decoration-color:
initial;" > </div>
<div dir="ltr" style="outline: none; color: rgb(18, 18, 18); font-family: Arial, Helvetica,
sans-serif; font-size: 14px; font-style: normal; font-variant-ligatures: normal; font-variant-caps:
normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent:
0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-stroke-width:
0px; background-color: rgb(255, 255, 255); text-decoration-style: initial; text-decoration-color:
initial;" >Some things I looked at:</div>
<div dir="ltr" style="outline: none; color: rgb(18, 18, 18); font-family: Arial, Helvetica,
sans-serif; font-size: 14px; font-style: normal; font-variant-ligatures: normal; font-variant-caps:
normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent:
0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-stroke-width:
0px; background-color: rgb(255, 255, 255); text-decoration-style: initial; text-decoration-color:
initial;" > </div>
<div dir="ltr" style="outline: none; color: rgb(18, 18, 18); font-family: Arial, Helvetica,
sans-serif; font-size: 14px; font-style: normal; font-variant-ligatures: normal; font-variant-caps:
normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent:
0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-stroke-width:
0px; background-color: rgb(255, 255, 255); text-decoration-style: initial; text-decoration-color:
initial;" ><div style="outline: none;" >`SPARK_PRINT_LAUNCH_COMMAND=1 /usr/local/spark-2.0.2-bin-hadoop2.7/bin/pyspark`</div>
<div style="outline: none;" ><br style="outline: none;" >Spark Command: python2.7<br
style="outline: none;" >========================================<br style="outline:
none;" >Python 2.7.13 |Anaconda custom (64-bit)| (default, Dec 20 2016, 23:09:15) <br
style="outline: none;" >[GCC 4.4.7 20120313 (Red Hat 4.4.7-1)] on linux2<br style="outline:
none;" >Type "help", "copyright", "credits" or "license" for more information.<br style="outline:
none;" >Anaconda is brought to you by Continuum Analytics.<br style="outline: none;"
>Please check out:<span> </span><a href="http://continuum.io/thanks"
rel="noopener" style="outline: none; position: relative; color: rgb(65, 120, 190);" target="_blank"
>http://continuum.io/thanks</a><span> </span>and<span> </span><a
href="https://anaconda.org/" rel="noopener" style="outline: none; position: relative; color:
rgb(65, 120, 190);" target="_blank" >https://anaconda.org</a><br style="outline:
none;" >Spark Command: **/usr/lib/jvm/java-8-openjdk-amd64/bin/java -cp /usr/local/spark/conf/**:/usr/local/spark/jars/*
-Xmx1g org.apache.spark.deploy.SparkSubmit --name PySparkShell pyspark-shell<br
style="outline: none;" > </div></div>
<div dir="ltr" style="outline: none; color: rgb(18, 18, 18); font-family: Arial, Helvetica,
sans-serif; font-size: 14px; font-style: normal; font-variant-ligatures: normal; font-variant-caps:
normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent:
0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-stroke-width:
0px; background-color: rgb(255, 255, 255); text-decoration-style: initial; text-decoration-color:
initial;" ><div style="outline: none;" ><div data-body-version="2b5fcc793ad7e136c07638a2c3c2dc9e"
id="issuecomment-5658136" style="outline: none;" ><div style="outline: none; box-sizing:
border-box;" ><table style="outline: none; box-sizing: border-box; border-spacing: 0px;
border-collapse: collapse; display: block !important;" > <tbody style="outline:
none; box-sizing: border-box; display: block !important;" > <tr style="outline:
none; box-sizing: border-box; display: block !important;" > <td
style="outline: none; box-sizing: border-box; padding: 15px; font-family: -apple-system, BlinkMacSystemFont,
"Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji",
"Segoe UI Emoji", "Segoe UI Symbol"; font-size: 14px;
line-height: 1.5; word-wrap: break-word; width: 698px; overflow: visible; display: block !important;"
> <div style="outline: none;" ><font face="Default Monospace,Courier
New,Courier,monospace" size="2" style="outline: none;" ><code style="outline: none;
box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono",
Menlo, Courier, monospace; font-size: 11.9px; padding: 0px; margin: 0px; background: transparent;
border-radius: 3px; word-break: normal; white-space: pre; border: 0px; display: inline; overflow:
visible; line-height: inherit; word-wrap: normal;" > PPID PID PGID
SID TTY TPGID STAT UID TIME
COMMAND<br style="outline: none;" > 0
1308 1308 1308 ?
1416 Ss 0 0:00 bash<br style="outline: none;"
> 1308 1416 1416 1308 ?
1416 R+ 0
0:00 \_ ps axjf<br style="outline: none;" >
0 1151 1151 1151 ?
1151 Ss+ 0 0:00 bash<br style="outline:
none;" > 0
1 1 1 ?
-1 Ss 0 0:00 /bin/bash /usr/local/bin/start-dsx-notebook.sh<br
style="outline: none;" > 1 1014
1 1 ?
-1 S 0 0:00 /bin/sh /user-home/.scripts/publishing-startup-scripts/nbexec_py_startup.sh<br
style="outline: none;" > 1014 1026
1 1 ?
-1 S 0 0:06 \_ python /user-home/.scripts/system/publishing-api/py2http.py<br
style="outline: none;" > 1 1017
1 1 ?
-1 S 0 0:00 su -l 1001 /usr/local/bin/start-user-notebook.sh
spark-master-svc:7077 dsx /user-home/1001/DSX_Projects/imagemgmt 1523893891668 imagemgmt Bearer
eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9.eyJ1c2VybmFtZSI6InRlc3QiLCJzdWIiOiJ0ZXN0IiwiaXNzIjoiS05PWFNTTyIsImF1ZCI6IkRTWCIsInJvbGUiOiJBZG1pbiIsInVpZCI6IjEwMDEiLCJpYXQiOjE1MjQ2MDIxMTd9.jHyjakD4G7XlOJ3Q1e5We3agHy_dtao_U98rZcLuTNBgGaETYKfHO2PC-94HG_nxIcTjDxymefWHItiwO7QcTIg_sIkP4uPSfQMTFthrMWNUucR0xRWJxFPcYgLlKo3T2P8JmA_LslVWqFD_MMjmYHI3UukVRj319_MSsRTW3Md3quF5mmv3OZMVjuI8faKMQF7zt_17W_QbNZAT91F0AboXJ7iazz71vcsuZZx0OxnSzJzcW3AEYb8JFWz3opbRwpc3dswbLco8TJ6I4DtacBq7syv3zg0bLIIcHSCp-LBwHrTyCWV7uJ0a3m-MSdvwdZ35WYE6_8LRwadKfW6hiw
1001<br style="outline: none;" > 1017 1018
1018 1018 ? -1
Ss 1001 0:00 \_ -su /usr/local/bin/start-user-notebook.sh
spark-master-svc:7077 dsx /user-home/1001/DSX_Projects/imagemgmt 1523893891668 imagemgmt Bearer
eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9.eyJ1c2VybmFtZSI6InRlc3QiLCJzdWIiOiJ0ZXN0IiwiaXNzIjoiS05PWFNTTyIsImF1ZCI6IkRTWCIsInJvbGUiOiJBZG1pbiIsInVpZCI6IjEwMDEiLCJpYXQiOjE1MjQ2MDIxMTd9.jHyjakD4G7XlOJ3Q1e5We3agHy_dtao_U98rZcLuTNBgGaETYKfHO2PC-94HG_nxIcTjDxymefWHItiwO7QcTIg_sIkP4uPSfQMTFthrMWNUucR0xRWJxFPcYgLlKo3T2P8JmA_LslVWqFD_MMjmYHI3UukVRj319_MSsRTW3Md3quF5mmv3OZMVjuI8faKMQF7zt_17W_QbNZAT91F0AboXJ7iazz71vcsuZZx0OxnSzJzcW3AEYb8JFWz3opbRwpc3dswbLco8TJ6I4DtacBq7syv3zg0bLIIcHSCp-LBwHrTyCWV7uJ0a3m-MSdvwdZ35WYE6_8LRwadKfW6hiw
1001<br style="outline: none;" > 1018 1025
1018 1018 ? -1
Sl 1001 0:51 \_ /opt/conda/bin/python
/opt/conda/bin/jupyter-notebook --NotebookApp.token= --port=8888 --no-browser<br style="outline:
none;" > 1025 1033 1033 1033
? -1 Ssl 1001
0:03 \_ python -m ipykernel_launcher
-f /user-home/1001/.local/share/jupyter/runtime/kernel-a11fc3e6-cf9d-4e6f-afe2-728ba48fb0bd.json<br
style="outline: none;" > 1033 1054 1033
1033 ? -1 Sl 1001
0:28 | \_ <strong
style="outline: none;" >/usr/lib/jvm/java-8-openjdk-amd64/jre/bin/java -cp /usr/local/spark/conf/:/usr/local/spark/jars/*
-Xmx1g -Djavax.net.ssl.trustStore=/user-home/_global_/security/customer-truststores/cacerts
org.apache.spark.deploy.SparkSubmit pyspark-shell</strong><br style="outline: none;"
> 1025 1167 1167 1167 ?
-1 Ssl 1001 2:07
\_ /usr/lib/jvm/java-8-openjdk-amd64/jre/bin/java
-cp /usr/local/spark/conf/:/usr/local/spark/jars/* -Xmx1g -Djavax.net.ssl.trustStore=/user-home/_global_/security/customer-truststores/cacerts
org.apache.spark.deploy.SparkSubmit --class org.apache.toree.Main /opt/conda/share/jupyter/kernels/apache_toree_scala/lib/toree-assembly-0.2.0.dev1-incubating-SNAPSHOT.jar
--profile /user-home/1001/.local/share/jupyter/runtime/kernel-1a4d3565-a32e-4d80-875b-93da83451a3c.json<br
style="outline: none;" > 1025 1318 1318
1318 ? -1 Ssl
1001 0:02 \_ /opt/conda/lib/R/bin/exec/R
--slave -e IRkernel::main() --args /user-home/1001/.local/share/jupyter/runtime/kernel-f037058c-013a-4eae-b554-4073585c9172.json<br
style="outline: none;" > 1 1328
1318 1318 ? -1
Sl 1001 0:09 /usr/lib/jvm/java-8-openjdk-amd64/jre/bin/java
-cp /usr/local/spark/conf/:/usr/local/spark/jars/* -Xmx1g -Djavax.net.ssl.trustStore=/user-home/_global_/security/customer-truststores/cacerts
org.apache.spark.deploy.SparkSubmit sparkr-shell /tmp/RtmpyKF5Yq/backend_port5266fef1bd1<br
style="outline: none;" > ~
</code></font></div>
</td> </tr> </tbody></table></div></div></div></div></div>
<div dir="ltr" > </div>
<div dir="ltr" ><div class="socmaildefaultfont" dir="ltr" style="font-family:Arial,
Helvetica, sans-serif;font-size:10.5pt" ><div class="socmaildefaultfont" dir="ltr" style="font-family:Arial;font-size:10.5pt"
><div class="socmaildefaultfont" dir="ltr" style="font-family:Arial;font-size:10.5pt"
><div dir="ltr" id="tpl_normal_20150824133417" style="font-family:arial,helvetica,sans-serif;font-size:9pt;"
>
<table border="0" cellpadding="0" cellspacing="0" style="border:0;table-layout:auto;font-family:
arial,helvetica,sans-serif;white-space:normal;width:650px;font-size:7pt" width="680px" >
<tbody> <tr> <td style="vertical-align:
bottom;" > <div style="font-size:9pt" ><span style="font-size:1.143em;"
><span style="font-family:Times New Roman,Times,serif;" >Regards,</span></span></div>
<div class="vcard" style="margin:8px 0 8px 0" ><span style="font-size:1.143em;"
><span style="font-family:Times New Roman,Times,serif;" ><b class="fn n" style="color:#888888;font-size:12pt"
><span class="given-name" >Mihai</span> <span class="family-name" >Iacob</span></b><br>
<a href="https://datascience.ibm.com/local" >DSX Local</a>
- Security, IBM Analytics</span></span></div>
</td> </tr> </tbody></table></div></div></div></div></div></div><BR>
---------------------------------------------------------------------
To unsubscribe e-mail: user-unsubscribe@spark.apache.org
|