spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From makoto <tokomakoma...@gmail.com>
Subject Re: pyspark configuration with Juyter
Date Sat, 04 Nov 2017 16:59:32 GMT
I setup environment variables in my ~/.bashrc as follows:

export PYSPARK_PYTHON=/usr/local/oss/anaconda3/bin/python3.6
export PYTHONPATH=$(ls -a
${SPARK_HOME}/python/lib/py4j-*-src.zip):${SPARK_HOME}/python:$PYTHONPATH
export PYSPARK_DRIVER_PYTHON=jupyter
export PYSPARK_DRIVER_PYTHON_OPTS='notebook'


2017-11-03 20:56 GMT+09:00 Jeff Zhang <zjffdu@gmail.com>:

>
> You are setting PYSPARK_DRIVER to jupyter, please set it to python exec
> file
>
>
> anudeep <anud33p@gmail.com>于2017年11月3日周五 下午7:31写道:
>
>> Hello experts,
>>
>> I install jupyter notebook thorugh anacoda, set the pyspark driver to use
>> jupyter notebook.
>>
>> I see the below issue when i try to open pyspark.
>>
>> anudeepg@datanode2 spark-2.1.0]$ ./bin/pyspark
>> [I 07:29:53.184 NotebookApp] The port 8888 is already in use, trying
>> another port.
>> [I 07:29:53.211 NotebookApp] JupyterLab alpha preview extension loaded
>> from /home/anudeepg/anaconda2/lib/python2.7/site-packages/jupyterlab
>> JupyterLab v0.27.0
>> Known labextensions:
>> [I 07:29:53.212 NotebookApp] Running the core application with no
>> additional extensions or settings
>> [I 07:29:53.214 NotebookApp] Serving notebooks from local directory:
>> /opt/mapr/spark/spark-2.1.0
>> [I 07:29:53.214 NotebookApp] 0 active kernels
>> [I 07:29:53.214 NotebookApp] The Jupyter Notebook is running at:
>> http://localhost:8889/?token=9aa5dc87cb5a6d987237f68e2f0b7e
>> 9c70a7f2e8c9a7cf2e
>> [I 07:29:53.214 NotebookApp] Use Control-C to stop this server and shut
>> down all kernels (twice to skip confirmation).
>> [W 07:29:53.214 NotebookApp] No web browser found: could not locate
>> runnable browser.
>> [C 07:29:53.214 NotebookApp]
>>
>>     Copy/paste this URL into your browser when you connect for the first
>> time,
>>     to login with a token:
>>         http://localhost:8889/?token=9aa5dc87cb5a6d987237f68e2f0b7e
>> 9c70a7f2e8c9a7cf2e
>>
>>
>> Can someone please help me here.
>>
>> Thanks!
>> Anudeep
>>
>>

Mime
View raw message