spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From nguyen duc tuan <newvalu...@gmail.com>
Subject Re: ImportError: No module named numpy
Date Thu, 02 Jun 2016 13:46:09 GMT
​​
You should set both PYSPARK_DRIVER_PYTHON and PYSPARK_PYTHON the path to
your python interpreter.

2016-06-02 20:32 GMT+07:00 Bhupendra Mishra <bhupendra.mishra@gmail.com>:

> did not resolved. :(
>
> On Thu, Jun 2, 2016 at 3:01 PM, Sergio Fernández <wikier@apache.org>
> wrote:
>
>>
>> On Thu, Jun 2, 2016 at 9:59 AM, Bhupendra Mishra <
>> bhupendra.mishra@gmail.com> wrote:
>>>
>>> and i have already exported environment variable in spark-env.sh as
>>> follows.. error still there  error: ImportError: No module named numpy
>>>
>>> export PYSPARK_PYTHON=/usr/bin/python
>>>
>>
>> According the documentation at
>> http://spark.apache.org/docs/latest/configuration.html#environment-variables
>> the PYSPARK_PYTHON environment variable is for poniting to the Python
>> interpreter binary.
>>
>> If you check the programming guide
>> https://spark.apache.org/docs/0.9.0/python-programming-guide.html#installing-and-configuring-pyspark
>> it says you need to add your custom path to PYTHONPATH (the script
>> automatically adds the bin/pyspark there).
>>
>> So typically in Linux you would need to add the following (assuming you
>> installed numpy there):
>>
>> export PYTHONPATH=$PYTHONPATH:/usr/lib/python2.7/dist-packages
>>
>> Hope that helps.
>>
>>
>>
>>
>>> On Thu, Jun 2, 2016 at 12:04 AM, Julio Antonio Soto de Vicente <
>>> julio@esbet.es> wrote:
>>>
>>>> Try adding to spark-env.sh (renaming if you still have it with
>>>> .template at the end):
>>>>
>>>> PYSPARK_PYTHON=/path/to/your/bin/python
>>>>
>>>> Where your bin/python is your actual Python environment with Numpy
>>>> installed.
>>>>
>>>>
>>>> El 1 jun 2016, a las 20:16, Bhupendra Mishra <
>>>> bhupendra.mishra@gmail.com> escribió:
>>>>
>>>> I have numpy installed but where I should setup PYTHONPATH?
>>>>
>>>>
>>>> On Wed, Jun 1, 2016 at 11:39 PM, Sergio Fernández <wikier@apache.org>
>>>> wrote:
>>>>
>>>>> sudo pip install numpy
>>>>>
>>>>> On Wed, Jun 1, 2016 at 5:56 PM, Bhupendra Mishra <
>>>>> bhupendra.mishra@gmail.com> wrote:
>>>>>
>>>>>> Thanks .
>>>>>> How can this be resolved?
>>>>>>
>>>>>> On Wed, Jun 1, 2016 at 9:02 PM, Holden Karau <holden@pigscanfly.ca>
>>>>>> wrote:
>>>>>>
>>>>>>> Generally this means numpy isn't installed on the system or your
>>>>>>> PYTHONPATH has somehow gotten pointed somewhere odd,
>>>>>>>
>>>>>>> On Wed, Jun 1, 2016 at 8:31 AM, Bhupendra Mishra <
>>>>>>> bhupendra.mishra@gmail.com> wrote:
>>>>>>>
>>>>>>>> If any one please can help me with following error.
>>>>>>>>
>>>>>>>>  File
>>>>>>>> "/opt/mapr/spark/spark-1.6.1/python/lib/pyspark.zip/pyspark/mllib/__init__.py",
>>>>>>>> line 25, in <module>
>>>>>>>>
>>>>>>>> ImportError: No module named numpy
>>>>>>>>
>>>>>>>>
>>>>>>>> Thanks in advance!
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Cell : 425-233-8271
>>>>>>> Twitter: https://twitter.com/holdenkarau
>>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Sergio Fernández
>>>>> Partner Technology Manager
>>>>> Redlink GmbH
>>>>> m: +43 6602747925
>>>>> e: sergio.fernandez@redlink.co
>>>>> w: http://redlink.co
>>>>>
>>>>
>>>>
>>>
>>
>>
>> --
>> Sergio Fernández
>> Partner Technology Manager
>> Redlink GmbH
>> m: +43 6602747925
>> e: sergio.fernandez@redlink.co
>> w: http://redlink.co
>>
>
>

Mime
View raw message