spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Kousuke Saruta (JIRA)" <j...@apache.org>
Subject [jira] [Created] (SPARK-28548) explain() shows wrong result for persisted DataFrames after some operations
Date Sun, 28 Jul 2019 18:43:00 GMT
Kousuke Saruta created SPARK-28548:
--------------------------------------

             Summary: explain() shows wrong result for persisted DataFrames after some operations
                 Key: SPARK-28548
                 URL: https://issues.apache.org/jira/browse/SPARK-28548
             Project: Spark
          Issue Type: Bug
          Components: SQL
    Affects Versions: 3.0.0
            Reporter: Kousuke Saruta
            Assignee: Kousuke Saruta


After some operations against Datasets and then persist them, Dataset.explain shows wrong
result.
One of those operations is explain() itself.
An example here.

{code}
val df = spark.range(10)
df.explain
df.persist
df.explain
{code}

Expected result is like as follows.
{code}
== Physical Plan ==
*(1) ColumnarToRow
+- InMemoryTableScan [id#7L]
      +- InMemoryRelation [id#7L], StorageLevel(disk, memory, deserialized, 1 replicas)
            +- *(1) Range (0, 10, step=1, splits=12)
{code}

But I got this.
{code}
== Physical Plan ==
*(1) Range (0, 10, step=1, splits=12)
{code}



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message