spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Gourav Sengupta <gourav.sengu...@gmail.com>
Subject Re: trouble with 'pip pyspark' pyspark.sql.functions. ³unresolved import² for col() and lit()
Date Wed, 04 Apr 2018 22:37:57 GMT
Hi,

the way I manage things is, download spark, and set SPARK_HOME and the
import findspark and run findspark.init(). And everything else works just
fine.

I have never tried pip install pyspark though.


Regards,
Gourav Sengupta

On Wed, Apr 4, 2018 at 11:28 PM, Andy Davidson <
Andy@santacruzintegration.com> wrote:

> I am having trouble setting up my python3 virtualenv.
>
> I created a virtualenv ‘spark-2.3.0’ Installed pyspark using pip how ever
> I am not able to import pyspark.sql.functions. I get “unresolved import”
> when I try to import col() and lit()
>
> from pyspark.sql.functions import *
>
>
> I found if I download spark from apache and set SPARK_ROOT I can get my
> juypter notebook to work. This is a very error prone work around. I am
> having simiilar problem with my eclipse pyDev virtualenv
>
> Any suggestions would be greatly appreciated
>
> Andy
>
>
> # pip show in virtualenv
>
> (spark-2.3.0) $ pip show pyspark
>
> Name: pyspark
>
> Version: 2.3.0
>
> Summary: Apache Spark Python API
>
> Home-page: https://github.com/apache/spark/tree/master/python
>
> Author: Spark Developers
>
> Author-email: dev@spark.apache.org
>
> License: http://www.apache.org/licenses/LICENSE-2.0
>
> Location: /Users/foo/workSpace/pythonEnv/spark-2.3.0/lib/
> python3.6/site-packages
>
> Requires: py4j
>
> (spark-2.3.0) $
>
> (spark-2.3.0) $ ls ~/workSpace/pythonEnv/spark-2.3.0/lib/python3.6/site-
> packages/pyspark/sql/functions.py
>
> ~/workSpace/pythonEnv/spark-2.3.0/lib/python3.6/site-packages/pyspark/sql/
> functions.py
>
>
> # Jupyter Notebook
>
> Export SPARK_ROOT=~/workSpace/spark/spark-2.3.0-bin-hadoop2.7
>
>
>
> Eclipse pyDev virtual ENV
>
>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe e-mail: user-unsubscribe@spark.apache.org
>

Mime
View raw message