spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Andy Davidson <A...@SantaCruzIntegration.com>
Subject trouble with 'pip pyspark' pyspark.sql.functions. ³unresolved import² for col() and lit()
Date Wed, 04 Apr 2018 22:28:04 GMT
I am having trouble setting up my python3 virtualenv.

I created a virtualenv Œspark-2.3.0¹ Installed pyspark using pip how ever I
am not able to import pyspark.sql.functions. I get ³unresolved import² when
I try to import col() and lit()

from pyspark.sql.functions import *


I found if I download spark from apache and set SPARK_ROOT I can get my
juypter notebook to work. This is a very error prone work around. I am
having simiilar problem with my eclipse pyDev virtualenv

Any suggestions would be greatly appreciated

Andy


# pip show in virtualenv

(spark-2.3.0) $ pip show pyspark

Name: pyspark

Version: 2.3.0

Summary: Apache Spark Python API

Home-page: https://github.com/apache/spark/tree/master/python

Author: Spark Developers

Author-email: dev@spark.apache.org

License: http://www.apache.org/licenses/LICENSE-2.0

Location: 
/Users/foo/workSpace/pythonEnv/spark-2.3.0/lib/python3.6/site-packages

Requires: py4j

(spark-2.3.0) $ 


(spark-2.3.0) $ ls 
~/workSpace/pythonEnv/spark-2.3.0/lib/python3.6/site-packages/pyspark/sql/fu
nctions.py 

~/workSpace/pythonEnv/spark-2.3.0/lib/python3.6/site-packages/pyspark/sql/fu
nctions.py


# Jupyter Notebook
Export SPARK_ROOT=~/workSpace/spark/spark-2.3.0-bin-hadoop2.7




Eclipse pyDev virtual ENV






Mime
View raw message