spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Judy Nash <judyn...@exchange.microsoft.com>
Subject RE: saveAsTable fails on Python with "Unresolved plan found"
Date Thu, 07 May 2015 14:38:17 GMT
SPARK-4825<https://issues.apache.org/jira/browse/SPARK-4825> looks like the right bug,
but it should've been fixed on 1.2.1.

Is a similar fix needed in Python?

From: Judy Nash
Sent: Thursday, May 7, 2015 7:26 AM
To: user@spark.apache.org
Subject: saveAsTable fails on Python with "Unresolved plan found"

Hello,

I am following the tutorial code on sql programming guide<https://spark.apache.org/docs/1.2.1/sql-programming-guide.html#inferring-the-schema-using-reflection>
to try out Python on spark 1.2.1.

SaveAsTable function works on Scala bur fails on python with "Unresolved plan found".

Broken Python code:

from pyspark.sql import SQLContext, Row

sqlContext = SQLContext(sc)

lines = sc.textFile("data.txt")

parts = lines.map(lambda l: l.split(","))

people = parts.map(lambda p: Row(id=p[0], name=p[1]))

schemaPeople = sqlContext.inferSchema(people)

schemaPeople.saveAsTable("peopletable")

saveAsTable fails with Unresolved plan found.
org.apache.spark.sql.catalyst.errors.package$TreeNodeException: Unresolved plan found, tree:
'CreateTableAsSelect None, pytable, false, None


This scala code works fine:

from pyspark.sql import SQLContext, Row

sqlContext = SQLContext(sc)

lines = sc.textFile("data.txt")

parts = lines.map(lambda l: l.split(","))

people = parts.map(lambda p: Row(id=p[0], name=p[1]))

schemaPeople = sqlContext.inferSchema(people)

schemaPeople.saveAsTable("peopletable")


Is this a known issue? Or am I not using Python correctly?

Thanks,
Judy

Mime
View raw message