spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Mihai Iacob" <mia...@ca.ibm.com>
Subject how to trace sparkDriver context creation for pyspark
Date Wed, 02 May 2018 14:39:31 GMT
<div class="socmaildefaultfont" dir="ltr" style="font-family:Arial, Helvetica, sans-serif;font-size:10.5pt"
><div dir="ltr" ><div dir="ltr" style="outline: none; color: rgb(18, 18, 18);
font-family: Arial, Helvetica, sans-serif; font-size: 14px; font-style: normal; font-variant-ligatures:
normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align:
start; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing:
0px; -webkit-text-stroke-width: 0px; background-color: rgb(255, 255, 255); text-decoration-style:
initial; text-decoration-color: initial;" >&nbsp;</div>
<div dir="ltr" style="outline: none; color: rgb(18, 18, 18); font-family: Arial, Helvetica,
sans-serif; font-size: 14px; font-style: normal; font-variant-ligatures: normal; font-variant-caps:
normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent:
0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-stroke-width:
0px; background-color: rgb(255, 255, 255); text-decoration-style: initial; text-decoration-color:
initial;" >&nbsp;</div>
<div dir="ltr" style="outline: none; color: rgb(18, 18, 18); font-family: Arial, Helvetica,
sans-serif; font-size: 14px; font-style: normal; font-variant-ligatures: normal; font-variant-caps:
normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent:
0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-stroke-width:
0px; background-color: rgb(255, 255, 255); text-decoration-style: initial; text-decoration-color:
initial;" >I have&nbsp;python jupyter&nbsp;notebook setup to create&nbsp;a
spark context by default, and<span>&nbsp;</span><strong style="outline:
none;" >sometimes</strong><span>&nbsp;</span>these fail with the
following error:</div>
<div dir="ltr" style="outline: none; color: rgb(18, 18, 18); font-family: Arial, Helvetica,
sans-serif; font-size: 14px; font-style: normal; font-variant-ligatures: normal; font-variant-caps:
normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent:
0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-stroke-width:
0px; background-color: rgb(255, 255, 255); text-decoration-style: initial; text-decoration-color:
initial;" >&nbsp;</div>
<div dir="ltr" style="outline: none; color: rgb(18, 18, 18); font-family: Arial, Helvetica,
sans-serif; font-size: 14px; font-style: normal; font-variant-ligatures: normal; font-variant-caps:
normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent:
0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-stroke-width:
0px; background-color: rgb(255, 255, 255); text-decoration-style: initial; text-decoration-color:
initial;" ><div style="outline: none;" ><font face="Default Monospace,Courier
New,Courier,monospace" size="2" style="outline: none;" ><code style="outline: none;
box-sizing: border-box; font-family: monospace, monospace; font-size: 1em;" >18/04/30 18:03:27
WARN Utils: Service 'sparkDriver' could not bind on port 0. Attempting port 1.<br style="outline:
none;" >18/04/30 18:03:27 ERROR SparkContext: Error initializing SparkContext.<br style="outline:
none;" >java.net.BindException: Cannot assign requested address: Service 'sparkDriver'
failed after 100 retries! Consider explicitly setting the appropriate port for the service
'sparkDriver' (for example spark.ui.port for SparkUI) to an available port or increasing spark.port.maxRetries.</code></font></div></div>
<div dir="ltr" style="outline: none; color: rgb(18, 18, 18); font-family: Arial, Helvetica,
sans-serif; font-size: 14px; font-style: normal; font-variant-ligatures: normal; font-variant-caps:
normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent:
0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-stroke-width:
0px; background-color: rgb(255, 255, 255); text-decoration-style: initial; text-decoration-color:
initial;" >I have tracked it down to two possible settings that may cause this in&nbsp;spark
2.0.2, client mode, standalone cluster setup, running in kubernetes:</div>
<div dir="ltr" style="outline: none; color: rgb(18, 18, 18); font-family: Arial, Helvetica,
sans-serif; font-size: 14px; font-style: normal; font-variant-ligatures: normal; font-variant-caps:
normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent:
0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-stroke-width:
0px; background-color: rgb(255, 255, 255); text-decoration-style: initial; text-decoration-color:
initial;" >&nbsp;</div>
<div dir="ltr" style="outline: none; color: rgb(18, 18, 18); font-family: Arial, Helvetica,
sans-serif; font-size: 14px; font-style: normal; font-variant-ligatures: normal; font-variant-caps:
normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent:
0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-stroke-width:
0px; background-color: rgb(255, 255, 255); text-decoration-style: initial; text-decoration-color:
initial;" >spark.driver.port - we don't set it, so it should be&nbsp;random</div>
<div dir="ltr" style="outline: none; color: rgb(18, 18, 18); font-family: Arial, Helvetica,
sans-serif; font-size: 14px; font-style: normal; font-variant-ligatures: normal; font-variant-caps:
normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent:
0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-stroke-width:
0px; background-color: rgb(255, 255, 255); text-decoration-style: initial; text-decoration-color:
initial;" >spark.ui.port - we set spark.ui.enabled=false so it should not try to bind to
this port.</div>
<div dir="ltr" style="outline: none; color: rgb(18, 18, 18); font-family: Arial, Helvetica,
sans-serif; font-size: 14px; font-style: normal; font-variant-ligatures: normal; font-variant-caps:
normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent:
0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-stroke-width:
0px; background-color: rgb(255, 255, 255); text-decoration-style: initial; text-decoration-color:
initial;" >&nbsp;</div>
<div dir="ltr" style="outline: none; color: rgb(18, 18, 18); font-family: Arial, Helvetica,
sans-serif; font-size: 14px; font-style: normal; font-variant-ligatures: normal; font-variant-caps:
normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent:
0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-stroke-width:
0px; background-color: rgb(255, 255, 255); text-decoration-style: initial; text-decoration-color:
initial;" >Short story is I do not know which one spark gets confused about, and looking
at spark code not clear how&nbsp;spark.ui.port would cause this even if the error message
lists it as a possible cause.</div>
<div dir="ltr" style="outline: none; color: rgb(18, 18, 18); font-family: Arial, Helvetica,
sans-serif; font-size: 14px; font-style: normal; font-variant-ligatures: normal; font-variant-caps:
normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent:
0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-stroke-width:
0px; background-color: rgb(255, 255, 255); text-decoration-style: initial; text-decoration-color:
initial;" >&nbsp;</div>
<div dir="ltr" style="outline: none; color: rgb(18, 18, 18); font-family: Arial, Helvetica,
sans-serif; font-size: 14px; font-style: normal; font-variant-ligatures: normal; font-variant-caps:
normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent:
0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-stroke-width:
0px; background-color: rgb(255, 255, 255); text-decoration-style: initial; text-decoration-color:
initial;" ><strong style="outline: none;" >Question 1:</strong><span>&nbsp;</span>have
you seen this before?</div>
<div dir="ltr" style="outline: none; color: rgb(18, 18, 18); font-family: Arial, Helvetica,
sans-serif; font-size: 14px; font-style: normal; font-variant-ligatures: normal; font-variant-caps:
normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent:
0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-stroke-width:
0px; background-color: rgb(255, 255, 255); text-decoration-style: initial; text-decoration-color:
initial;" ><strong style="outline: none;" >Question 2:</strong><span>&nbsp;</span>how
do&nbsp;I trace the spark driver process? It seems that I can only set the sc.logLevel
after the spark context is created, but I need to trace before the spark context is created.</div>
<div dir="ltr" style="outline: none; color: rgb(18, 18, 18); font-family: Arial, Helvetica,
sans-serif; font-size: 14px; font-style: normal; font-variant-ligatures: normal; font-variant-caps:
normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent:
0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-stroke-width:
0px; background-color: rgb(255, 255, 255); text-decoration-style: initial; text-decoration-color:
initial;" >&nbsp;</div>
<div dir="ltr" style="outline: none; color: rgb(18, 18, 18); font-family: Arial, Helvetica,
sans-serif; font-size: 14px; font-style: normal; font-variant-ligatures: normal; font-variant-caps:
normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent:
0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-stroke-width:
0px; background-color: rgb(255, 255, 255); text-decoration-style: initial; text-decoration-color:
initial;" >I created a log4j.properties file in the spark/conf directory and set it to
TRACE but that only gets picked up when I run a Scala jupyter&nbsp;notebook, not when
I run a python juypyter notebook, and I haven't been able to find out how to turn the same
level of tracing for a spark-driver process started via a python jupyter notebook.</div>
<div dir="ltr" style="outline: none; color: rgb(18, 18, 18); font-family: Arial, Helvetica,
sans-serif; font-size: 14px; font-style: normal; font-variant-ligatures: normal; font-variant-caps:
normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent:
0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-stroke-width:
0px; background-color: rgb(255, 255, 255); text-decoration-style: initial; text-decoration-color:
initial;" >&nbsp;</div>
<div dir="ltr" style="outline: none; color: rgb(18, 18, 18); font-family: Arial, Helvetica,
sans-serif; font-size: 14px; font-style: normal; font-variant-ligatures: normal; font-variant-caps:
normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent:
0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-stroke-width:
0px; background-color: rgb(255, 255, 255); text-decoration-style: initial; text-decoration-color:
initial;" >Some things I looked at:</div>
<div dir="ltr" style="outline: none; color: rgb(18, 18, 18); font-family: Arial, Helvetica,
sans-serif; font-size: 14px; font-style: normal; font-variant-ligatures: normal; font-variant-caps:
normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent:
0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-stroke-width:
0px; background-color: rgb(255, 255, 255); text-decoration-style: initial; text-decoration-color:
initial;" >&nbsp;</div>
<div dir="ltr" style="outline: none; color: rgb(18, 18, 18); font-family: Arial, Helvetica,
sans-serif; font-size: 14px; font-style: normal; font-variant-ligatures: normal; font-variant-caps:
normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent:
0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-stroke-width:
0px; background-color: rgb(255, 255, 255); text-decoration-style: initial; text-decoration-color:
initial;" ><div style="outline: none;" >`SPARK_PRINT_LAUNCH_COMMAND=1 /usr/local/spark-2.0.2-bin-hadoop2.7/bin/pyspark`</div>
<div style="outline: none;" ><br style="outline: none;" >Spark Command: python2.7<br
style="outline: none;" >========================================<br style="outline:
none;" >Python 2.7.13 |Anaconda custom (64-bit)| (default, Dec 20 2016, 23:09:15)&nbsp;<br
style="outline: none;" >[GCC 4.4.7 20120313 (Red Hat 4.4.7-1)] on linux2<br style="outline:
none;" >Type "help", "copyright", "credits" or "license" for more information.<br style="outline:
none;" >Anaconda is brought to you by Continuum Analytics.<br style="outline: none;"
>Please check out:<span>&nbsp;</span><a href="http://continuum.io/thanks"
rel="noopener" style="outline: none; position: relative; color: rgb(65, 120, 190);" target="_blank"
>http://continuum.io/thanks</a><span>&nbsp;</span>and<span>&nbsp;</span><a
href="https://anaconda.org/" rel="noopener" style="outline: none; position: relative; color:
rgb(65, 120, 190);" target="_blank" >https://anaconda.org</a><br style="outline:
none;" >Spark Command: **/usr/lib/jvm/java-8-openjdk-amd64/bin/java -cp /usr/local/spark/conf/**:/usr/local/spark/jars/*
-Xmx1g&nbsp;&nbsp;org.apache.spark.deploy.SparkSubmit --name PySparkShell pyspark-shell<br
style="outline: none;" >&nbsp;</div></div>
<div dir="ltr" style="outline: none; color: rgb(18, 18, 18); font-family: Arial, Helvetica,
sans-serif; font-size: 14px; font-style: normal; font-variant-ligatures: normal; font-variant-caps:
normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent:
0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-stroke-width:
0px; background-color: rgb(255, 255, 255); text-decoration-style: initial; text-decoration-color:
initial;" ><div style="outline: none;" ><div data-body-version="2b5fcc793ad7e136c07638a2c3c2dc9e"
id="issuecomment-5658136" style="outline: none;" ><div style="outline: none; box-sizing:
border-box;" ><table style="outline: none; box-sizing: border-box; border-spacing: 0px;
border-collapse: collapse; display: block !important;" >        <tbody style="outline:
none; box-sizing: border-box; display: block !important;" >                <tr style="outline:
none; box-sizing: border-box; display: block !important;" >                        <td
style="outline: none; box-sizing: border-box; padding: 15px; font-family: -apple-system, BlinkMacSystemFont,
&quot;Segoe UI&quot;, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;,
&quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;; font-size: 14px;
line-height: 1.5; word-wrap: break-word; width: 698px; overflow: visible; display: block !important;"
>                        <div style="outline: none;" ><font face="Default Monospace,Courier
New,Courier,monospace" size="2" style="outline: none;" ><code style="outline: none;
box-sizing: border-box; font-family: SFMono-Regular, Consolas, &quot;Liberation Mono&quot;,
Menlo, Courier, monospace; font-size: 11.9px; padding: 0px; margin: 0px; background: transparent;
border-radius: 3px; word-break: normal; white-space: pre; border: 0px; display: inline; overflow:
visible; line-height: inherit; word-wrap: normal;" >&nbsp;PPID &nbsp; PID &nbsp;PGID
&nbsp; SID TTY &nbsp; &nbsp; &nbsp;TPGID STAT &nbsp; UID &nbsp; TIME
COMMAND<br style="outline: none;" >                        &nbsp;&nbsp; &nbsp;0
&nbsp;1308 &nbsp;1308 &nbsp;1308 ? &nbsp; &nbsp; &nbsp; &nbsp;
1416 Ss &nbsp; &nbsp; &nbsp; 0 &nbsp; 0:00 bash<br style="outline: none;"
>                        &nbsp;1308 &nbsp;1416 &nbsp;1416 &nbsp;1308 ?
&nbsp; &nbsp; &nbsp; &nbsp; 1416 R+ &nbsp; &nbsp; &nbsp; 0 &nbsp;
0:00 &nbsp;\_ ps axjf<br style="outline: none;" >                        &nbsp;&nbsp;
&nbsp;0 &nbsp;1151 &nbsp;1151 &nbsp;1151 ? &nbsp; &nbsp; &nbsp;
&nbsp; 1151 Ss+ &nbsp; &nbsp; &nbsp;0 &nbsp; 0:00 bash<br style="outline:
none;" >                        &nbsp;&nbsp; &nbsp;0 &nbsp; &nbsp;
1 &nbsp; &nbsp; 1 &nbsp; &nbsp; 1 ? &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; -1 Ss &nbsp; &nbsp; &nbsp; 0 &nbsp; 0:00 /bin/bash /usr/local/bin/start-dsx-notebook.sh<br
style="outline: none;" >                        &nbsp;&nbsp; &nbsp;1 &nbsp;1014
&nbsp; &nbsp; 1 &nbsp; &nbsp; 1 ? &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; -1 S &nbsp; &nbsp; &nbsp; &nbsp;0 &nbsp; 0:00 /bin/sh /user-home/.scripts/publishing-startup-scripts/nbexec_py_startup.sh<br
style="outline: none;" >                        &nbsp;1014 &nbsp;1026 &nbsp;
&nbsp; 1 &nbsp; &nbsp; 1 ? &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
-1 S &nbsp; &nbsp; &nbsp; &nbsp;0 &nbsp; 0:06 &nbsp;\_ python /user-home/.scripts/system/publishing-api/py2http.py<br
style="outline: none;" >                        &nbsp;&nbsp; &nbsp;1 &nbsp;1017
&nbsp; &nbsp; 1 &nbsp; &nbsp; 1 ? &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; -1 S &nbsp; &nbsp; &nbsp; &nbsp;0 &nbsp; 0:00 su -l 1001 /usr/local/bin/start-user-notebook.sh
spark-master-svc:7077 dsx /user-home/1001/DSX_Projects/imagemgmt 1523893891668 imagemgmt Bearer
eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9.eyJ1c2VybmFtZSI6InRlc3QiLCJzdWIiOiJ0ZXN0IiwiaXNzIjoiS05PWFNTTyIsImF1ZCI6IkRTWCIsInJvbGUiOiJBZG1pbiIsInVpZCI6IjEwMDEiLCJpYXQiOjE1MjQ2MDIxMTd9.jHyjakD4G7XlOJ3Q1e5We3agHy_dtao_U98rZcLuTNBgGaETYKfHO2PC-94HG_nxIcTjDxymefWHItiwO7QcTIg_sIkP4uPSfQMTFthrMWNUucR0xRWJxFPcYgLlKo3T2P8JmA_LslVWqFD_MMjmYHI3UukVRj319_MSsRTW3Md3quF5mmv3OZMVjuI8faKMQF7zt_17W_QbNZAT91F0AboXJ7iazz71vcsuZZx0OxnSzJzcW3AEYb8JFWz3opbRwpc3dswbLco8TJ6I4DtacBq7syv3zg0bLIIcHSCp-LBwHrTyCWV7uJ0a3m-MSdvwdZ35WYE6_8LRwadKfW6hiw
1001<br style="outline: none;" >                        &nbsp;1017 &nbsp;1018
&nbsp;1018 &nbsp;1018 ? &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; -1
Ss &nbsp; &nbsp;1001 &nbsp; 0:00 &nbsp;\_ -su /usr/local/bin/start-user-notebook.sh
spark-master-svc:7077 dsx /user-home/1001/DSX_Projects/imagemgmt 1523893891668 imagemgmt Bearer
eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9.eyJ1c2VybmFtZSI6InRlc3QiLCJzdWIiOiJ0ZXN0IiwiaXNzIjoiS05PWFNTTyIsImF1ZCI6IkRTWCIsInJvbGUiOiJBZG1pbiIsInVpZCI6IjEwMDEiLCJpYXQiOjE1MjQ2MDIxMTd9.jHyjakD4G7XlOJ3Q1e5We3agHy_dtao_U98rZcLuTNBgGaETYKfHO2PC-94HG_nxIcTjDxymefWHItiwO7QcTIg_sIkP4uPSfQMTFthrMWNUucR0xRWJxFPcYgLlKo3T2P8JmA_LslVWqFD_MMjmYHI3UukVRj319_MSsRTW3Md3quF5mmv3OZMVjuI8faKMQF7zt_17W_QbNZAT91F0AboXJ7iazz71vcsuZZx0OxnSzJzcW3AEYb8JFWz3opbRwpc3dswbLco8TJ6I4DtacBq7syv3zg0bLIIcHSCp-LBwHrTyCWV7uJ0a3m-MSdvwdZ35WYE6_8LRwadKfW6hiw
1001<br style="outline: none;" >                        &nbsp;1018 &nbsp;1025
&nbsp;1018 &nbsp;1018 ? &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; -1
Sl &nbsp; &nbsp;1001 &nbsp; 0:51 &nbsp; &nbsp; &nbsp;\_ /opt/conda/bin/python
/opt/conda/bin/jupyter-notebook --NotebookApp.token= --port=8888 --no-browser<br style="outline:
none;" >                        &nbsp;1025 &nbsp;1033 &nbsp;1033 &nbsp;1033
? &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; -1 Ssl &nbsp; 1001 &nbsp;
0:03 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;\_ python -m ipykernel_launcher
-f /user-home/1001/.local/share/jupyter/runtime/kernel-a11fc3e6-cf9d-4e6f-afe2-728ba48fb0bd.json<br
style="outline: none;" >                        &nbsp;1033 &nbsp;1054 &nbsp;1033
&nbsp;1033 ? &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; -1 Sl &nbsp; &nbsp;1001
&nbsp; 0:28 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;| &nbsp; \_ <strong
style="outline: none;" >/usr/lib/jvm/java-8-openjdk-amd64/jre/bin/java -cp /usr/local/spark/conf/:/usr/local/spark/jars/*
-Xmx1g -Djavax.net.ssl.trustStore=/user-home/_global_/security/customer-truststores/cacerts
org.apache.spark.deploy.SparkSubmit pyspark-shell</strong><br style="outline: none;"
>                        &nbsp;1025 &nbsp;1167 &nbsp;1167 &nbsp;1167 ?
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; -1 Ssl &nbsp; 1001 &nbsp; 2:07
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp;\_ /usr/lib/jvm/java-8-openjdk-amd64/jre/bin/java
-cp /usr/local/spark/conf/:/usr/local/spark/jars/* -Xmx1g -Djavax.net.ssl.trustStore=/user-home/_global_/security/customer-truststores/cacerts
org.apache.spark.deploy.SparkSubmit --class org.apache.toree.Main /opt/conda/share/jupyter/kernels/apache_toree_scala/lib/toree-assembly-0.2.0.dev1-incubating-SNAPSHOT.jar
--profile /user-home/1001/.local/share/jupyter/runtime/kernel-1a4d3565-a32e-4d80-875b-93da83451a3c.json<br
style="outline: none;" >                        &nbsp;1025 &nbsp;1318 &nbsp;1318
&nbsp;1318 ? &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; -1 Ssl &nbsp;
1001 &nbsp; 0:02 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;\_ /opt/conda/lib/R/bin/exec/R
--slave -e IRkernel::main() --args /user-home/1001/.local/share/jupyter/runtime/kernel-f037058c-013a-4eae-b554-4073585c9172.json<br
style="outline: none;" >                        &nbsp;&nbsp; &nbsp;1 &nbsp;1328
&nbsp;1318 &nbsp;1318 ? &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; -1
Sl &nbsp; &nbsp;1001 &nbsp; 0:09 /usr/lib/jvm/java-8-openjdk-amd64/jre/bin/java
-cp /usr/local/spark/conf/:/usr/local/spark/jars/* -Xmx1g -Djavax.net.ssl.trustStore=/user-home/_global_/security/customer-truststores/cacerts
org.apache.spark.deploy.SparkSubmit sparkr-shell /tmp/RtmpyKF5Yq/backend_port5266fef1bd1<br
style="outline: none;" >                        ~ &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; </code></font></div>
                       </td>                </tr>        </tbody></table></div></div></div></div></div>
<div dir="ltr" >&nbsp;</div>
<div dir="ltr" ><div class="socmaildefaultfont" dir="ltr" style="font-family:Arial,
Helvetica, sans-serif;font-size:10.5pt" ><div class="socmaildefaultfont" dir="ltr" style="font-family:Arial;font-size:10.5pt"
><div class="socmaildefaultfont" dir="ltr" style="font-family:Arial;font-size:10.5pt"
><div dir="ltr" id="tpl_normal_20150824133417" style="font-family:arial,helvetica,sans-serif;font-size:9pt;"
>&nbsp;
<table border="0" cellpadding="0" cellspacing="0" style="border:0;table-layout:auto;font-family:
arial,helvetica,sans-serif;white-space:normal;width:650px;font-size:7pt" width="680px" >
       <tbody>                <tr>                        <td style="vertical-align:
bottom;" >                        <div style="font-size:9pt" ><span style="font-size:1.143em;"
><span style="font-family:Times New Roman,Times,serif;" >Regards,</span></span></div>
                       &nbsp;

                        <div class="vcard" style="margin:8px 0 8px 0" ><span style="font-size:1.143em;"
><span style="font-family:Times New Roman,Times,serif;" ><b class="fn n" style="color:#888888;font-size:12pt"
><span class="given-name" >Mihai</span> <span class="family-name" >Iacob</span></b><br>
                       <a href="https://datascience.ibm.com/local" >DSX&nbsp;Local</a>
- Security,&nbsp;IBM Analytics</span></span></div>                 
      </td>                </tr>        </tbody></table></div></div></div></div></div></div><BR>


---------------------------------------------------------------------
To unsubscribe e-mail: user-unsubscribe@spark.apache.org


Mime
View raw message