spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Zerony Zhao <bw.li...@gmail.com>
Subject Re: PYSPARK_DRIVER_PYTHON="ipython" spark/bin/pyspark Does not create SparkContext
Date Mon, 27 Jul 2015 21:30:19 GMT
Thank you so much.

I found the issue. My fault, the stock ipython version 0.12.1 is too old,
which does not support PYTHONSTARTUP. Upgrading ipython solved the issue.

On Mon, Jul 27, 2015 at 12:43 PM, <felixcheung_m@hotmail.com> wrote:

>  Hmm, it should work with you run `PYSPARK_DRIVER_PYTHON="ipython"
> spark/bin/pyspark`
>
> PYTHONSTARTUP is a PYTHON environment variable
>
> https <https://docs.python.org/2/using/cmdline.html#envvar-PYTHONSTARTUP>
> :// <https://docs.python.org/2/using/cmdline.html#envvar-PYTHONSTARTUP>
> docs.python.org
> <https://docs.python.org/2/using/cmdline.html#envvar-PYTHONSTARTUP>
> /2/using/
> <https://docs.python.org/2/using/cmdline.html#envvar-PYTHONSTARTUP>
> cmdline.html#envvar-PYTHONSTARTUP
> <https://docs.python.org/2/using/cmdline.html#envvar-PYTHONSTARTUP>
>
>
>
> On Sun, Jul 26, 2015 at 4:06 PM -0700, "Zerony Zhao" <bw.linux@gmail.com>
> wrote:
>
>     Hello everyone,
>
>  I have a newbie question.
>
>  $SPARK_HOME/bin/pyspark will create SparkContext automatically.
>
> Welcome to
>       ____              __
>      / __/__  ___ _____/ /__
>     _\ \/ _ \/ _ `/ __/  '_/
>    /__ / .__/\_,_/_/ /_/\_\   version 1.4.1
>       /_/
>
> Using Python version 2.7.3 (default, Jun 22 2015 19:33:41)
> SparkContext available as sc, HiveContext available as sqlContext.
>
>
>  But When using ipython as a driver,
>
> PYSPARK_DRIVER_PYTHON="ipython" spark/bin/pyspark
>
>  , does not create SparkContext automatically. I have to execute
>
> execfile('spark_home/python/pyspark/shell.py')
>
>  is it by design?
>
>  I read the bash script bin/pyspark, I noticed the line:
>
> export PYTHONSTARTUP="$SPARK_HOME/python/pyspark/shell.py"
>
>  But I searched the whole spark source code, the variable PYTHONSTARTUP
> is never used, I could not understand when PYTHONSTARTUP is executed.
>
>  Thank you.
>

Mime
View raw message