spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Kabeer Ahmed" <>
Subject Re: Setting conf options in jupyter
Date Sun, 02 Oct 2016 22:55:25 GMT
<html><head></head><body><div style="font-family: Verdana;font-size:


<div>Try something based on the below lines. You should get rid of the error that you
have reported.</div>




<div>scala&gt; sc.stop</div>


<div>scala&gt; :paste</div>

<div>// Entering paste mode (ctrl-D to finish)</div>


<div>import org.apache.spark._</div>

<div>val sc = new SparkContext(</div>

<div>&nbsp; &nbsp; &nbsp; new SparkConf().setAppName(&quot;bar&quot;)</div>

<div>&nbsp; &nbsp; &nbsp; )</div>

<div name="quote" style="margin:10px 5px 5px 10px; padding: 10px 0 10px 10px; border-left:2px
solid #C3D9E5; word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;">
<div style="margin:0 0 10px 0;"><b>Sent:</b>&nbsp;Thursday, September
29, 2016 at 9:23 PM<br/>
<b>From:</b>&nbsp;&quot;William Kupersanin&quot; &lt;;<br/>
<b>Subject:</b>&nbsp;Setting conf options in jupyter</div>

<div name="quoted-content">

<div>I am trying to figure out how to correctly set config options in jupyter when I
am already provided a SparkContext and a HiveContext. I need to increase a couple of memory
allocations. My program dies indicating that I am trying to call methods on a stopped SparkContext.
I thought I had created a new one with the new conf so I am not sure why&nbsp;</div>


<div>My code is as follows:</div>


<div>from pyspark import SparkConf, SparkContext</div>

<div>from pyspark.sql import HiveContext</div>

<div>from pyspark.sql import SQLContext</div>

<div>conf = (SparkConf()</div>

<div>&nbsp; &nbsp; &nbsp; &nbsp; .set(&quot;spark.yarn.executor.memoryOverhead&quot;,

<div>&nbsp; &nbsp; &nbsp; &nbsp;.set(&quot;spark.kryoserializer.buffer.max.mb&quot;,



<div>sc = SparkContext(conf=conf)</div>

<div>sqlContext2 = SQLContext.getOrCreate(sc)</div>

<div>starttime = time.time()</div>

<div>sampledate = &quot;20160913&quot;</div>

<div>networkdf =;/sp/network/&quot; + sampledate
+ &quot;/03/*&quot;)</div>



<div>An error occurred while calling o144.json.</div>

<div>: java.lang.IllegalStateException: Cannot call methods on a stopped SparkContext.</div>

<div>This stopped SparkContext was created at:....</div>

To unsubscribe e-mail:

View raw message