spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Nicholas Chammas (JIRA)" <>
Subject [jira] [Created] (SPARK-2797) SchemaRDDs don't support unpersist()
Date Fri, 01 Aug 2014 21:57:38 GMT
Nicholas Chammas created SPARK-2797:

             Summary: SchemaRDDs don't support unpersist()
                 Key: SPARK-2797
             Project: Spark
          Issue Type: Bug
          Components: PySpark, SQL
    Affects Versions: 1.0.2
            Reporter: Nicholas Chammas

Looks like something simple got missed in the Java layer?

>>> from pyspark.sql import SQLContext
>>> sqlContext = SQLContext(sc)
>>> raw = sc.parallelize(['{"a": 5}'])
>>> events = sqlContext.jsonRDD(raw)
>>> events.printSchema()
 |-- a: IntegerType

>>> events.cache()
PythonRDD[45] at RDD at PythonRDD.scala:37
>>> events.unpersist()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/root/spark/python/pyspark/", line 440, in unpersist
  File "/root/spark/python/lib/", line 537, in __call__
  File "/root/spark/python/lib/", line 304, in get_return_value
py4j.protocol.Py4JError: An error occurred while calling o108.unpersist. Trace:
py4j.Py4JException: Method unpersist([]) does not exist
	at py4j.reflection.ReflectionEngine.getMethod(
	at py4j.reflection.ReflectionEngine.getMethod(
	at py4j.Gateway.invoke(
	at py4j.commands.AbstractCommand.invokeMethod(
	at py4j.commands.CallCommand.execute(
>>> events.unpersist
<bound method SchemaRDD.unpersist of PythonRDD[45] at RDD at PythonRDD.scala:37>

This is on {{1.0.2-rc1}}.

This message was sent by Atlassian JIRA

View raw message