spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Attila Csordas <attilacsor...@gmail.com>
Subject Re: Using Python IDE for Spark Application Development
Date Sun, 04 Jan 2015 22:05:30 GMT
Dear All,

I tried to use the gist
https://gist.github.com/bigaidream/40fe0f8267a80e7c9cf8 mentioned earlier
and also added spark path
to the content root as suggested.

but I am running into this problem of things working with the python shell
supposedly but not with python standalone executable mode or something
along these lines:
http://stackoverflow.com/questions/11566967/python-raise-child-exception-oserror-errno-2-no-such-file-or-directory

I get a child_exception = [Errno 2] No such file or directory and setting
'Shell = True' does not solve the problem

see screenshot attached. On the other hand setting 'Shell = True' is not a
recommended practice in general:
http://stackoverflow.com/questions/3172470/actual-meaning-of-shell-true-in-subprocess

Any help is appreciated.

Cheers,
-- 
Attila Csordas
Bioinformatician
European Bioinformatics Institute, Hinxton, UK
Twitter: @attilacsordas


On Thu, Aug 14, 2014 at 2:41 AM, Sathish Kumaran Vairavelu <
vsathishkumaran@gmail.com> wrote:

> Thank you. It works
>
>
> On Wed, Aug 13, 2014 at 3:17 PM, smyee@yahoo.com <smyee@yahoo.com> wrote:
>
>> Satish, saw your message on the archiveā€¦. probably figured out by now.
>> To be able to work with pyspark in PyCharm console, you need to add the
>> spark path to the Content Root
>>
>> Take a look at this gisthttps://gist.github.com/bigaidream/40fe0f8267a80e7c9cf8
>> That worked for me.
>>
>>
>> On Wed, Aug 6, 2014 at 7:32 PM, Sathish Kumaran Vairavelu <vsathishkumaran@gmail.com>
wrote:
>>
>> > Mohit, This doesn't seems to be working can you please provide more
>> > details? when I use "from pyspark import SparkContext" it is disabled in
>> > pycharm. I use pycharm community edition. Where should I set the
>> > environment variables in same python script or different python script?
>> >
>> > Also, should I run any Spark local cluster so Spark program runs on top of
>> > that?
>> >
>> >
>> > Appreciate your help
>> >
>> > -Sathish
>> >
>> >
>> > On Wed, Aug 6, 2014 at 6:22 PM, Mohit Singh <mohit1007@gmail.com> wrote:
>> >
>> >> My naive set up..
>> >> Adding
>> >> os.environ['SPARK_HOME'] = "/path/to/spark"
>> >> sys.path.append("/path/to/spark/python")
>> >> on top of my script.
>> >> from pyspark import SparkContext
>> >> from pyspark import SparkConf
>> >> Execution works from within pycharm...
>> >>
>> >> Though my next step is to figure out autocompletion and I bet there are
>> >> better ways to develop apps for spark..
>> >>
>> >>
>> >>
>> >> On Wed, Aug 6, 2014 at 4:16 PM, Sathish Kumaran Vairavelu <
>> >> vsathishkumaran@gmail.com> wrote:
>> >>
>> >>> Hello,
>> >>>
>> >>> I am trying to use the python IDE PyCharm for Spark application
>> >>> development. How can I use pyspark with Python IDE? Can anyone help
me with
>> >>> this?
>> >>>
>> >>>
>> >>> Thanks
>> >>>
>> >>> Sathish
>>
>>

Mime
View raw message