spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sathish Kumaran Vairavelu <vsathishkuma...@gmail.com>
Subject Re: Using Python IDE for Spark Application Development
Date Thu, 14 Aug 2014 01:41:26 GMT
Thank you. It works


On Wed, Aug 13, 2014 at 3:17 PM, smyee@yahoo.com <smyee@yahoo.com> wrote:

> Satish, saw your message on the archiveā€¦. probably figured out by now.
> To be able to work with pyspark in PyCharm console, you need to add the
> spark path to the Content Root
>
> Take a look at this gisthttps://gist.github.com/bigaidream/40fe0f8267a80e7c9cf8
> That worked for me.
>
>
> On Wed, Aug 6, 2014 at 7:32 PM, Sathish Kumaran Vairavelu <vsathishkumaran@gmail.com>
wrote:
>
> > Mohit, This doesn't seems to be working can you please provide more
> > details? when I use "from pyspark import SparkContext" it is disabled in
> > pycharm. I use pycharm community edition. Where should I set the
> > environment variables in same python script or different python script?
> >
> > Also, should I run any Spark local cluster so Spark program runs on top of
> > that?
> >
> >
> > Appreciate your help
> >
> > -Sathish
> >
> >
> > On Wed, Aug 6, 2014 at 6:22 PM, Mohit Singh <mohit1007@gmail.com> wrote:
> >
> >> My naive set up..
> >> Adding
> >> os.environ['SPARK_HOME'] = "/path/to/spark"
> >> sys.path.append("/path/to/spark/python")
> >> on top of my script.
> >> from pyspark import SparkContext
> >> from pyspark import SparkConf
> >> Execution works from within pycharm...
> >>
> >> Though my next step is to figure out autocompletion and I bet there are
> >> better ways to develop apps for spark..
> >>
> >>
> >>
> >> On Wed, Aug 6, 2014 at 4:16 PM, Sathish Kumaran Vairavelu <
> >> vsathishkumaran@gmail.com> wrote:
> >>
> >>> Hello,
> >>>
> >>> I am trying to use the python IDE PyCharm for Spark application
> >>> development. How can I use pyspark with Python IDE? Can anyone help me with
> >>> this?
> >>>
> >>>
> >>> Thanks
> >>>
> >>> Sathish
> >>>
> >>>
> >>>
> >>
> >>
> >> --
> >> Mohit
> >>
> >> "When you want success as badly as you want the air, then you will get
> >> it. There is no other secret of success."
> >> -Socrates
> >>
> >
> >
>
>
> --
> Mohit
>
> "When you want success as badly as you want the air, then you will get it.
> There is no other secret of success."
> -Socrates
>
>
>

Mime
View raw message