spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Yanbo Liang (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (SPARK-5926) [SQL] DataFrame.explain() return false result for DDL command
Date Fri, 20 Feb 2015 18:07:13 GMT

    [ https://issues.apache.org/jira/browse/SPARK-5926?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14329269#comment-14329269
] 

Yanbo Liang edited comment on SPARK-5926 at 2/20/15 6:06 PM:
-------------------------------------------------------------

This is because that for DDL like queries with side effects, and DataFrame force it happen
right away. We should use the former queryExecution.logical to explain.
{code:title=DataFrameImpl.scala|borderStyle=solid} 
@transient protected[sql] override lazy val logicalPlan: LogicalPlan = queryExecution.logical
match {
    // For various commands (like DDL) and queries with side effects, we force query optimization
to
    // happen right away to let these side effects take place eagerly.
    case _: Command |
         _: InsertIntoTable |
         _: CreateTableAsSelect[_] |
         _: CreateTableUsingAsSelect |
         _: WriteToFile =>
      LogicalRDD(queryExecution.analyzed.output, queryExecution.toRdd)(sqlContext)
    case _ =>
      queryExecution.logical
  }
{code} 


was (Author: yanboliang):
This is because that in DataFrameImpl 
{code:title=DataFrameImpl.scala|borderStyle=solid} 
@transient protected[sql] override lazy val logicalPlan: LogicalPlan = queryExecution.logical
match {
    // For various commands (like DDL) and queries with side effects, we force query optimization
to
    // happen right away to let these side effects take place eagerly.
    case _: Command |
         _: InsertIntoTable |
         _: CreateTableAsSelect[_] |
         _: CreateTableUsingAsSelect |
         _: WriteToFile =>
      LogicalRDD(queryExecution.analyzed.output, queryExecution.toRdd)(sqlContext)
    case _ =>
      queryExecution.logical
  }
{code} 

> [SQL] DataFrame.explain() return false result for DDL command
> -------------------------------------------------------------
>
>                 Key: SPARK-5926
>                 URL: https://issues.apache.org/jira/browse/SPARK-5926
>             Project: Spark
>          Issue Type: Bug
>          Components: SQL
>            Reporter: Yanbo Liang
>
> This bug is easy to reproduce, the following two queries should print out the same explain
result, but it's not.
> sql("create table tb as select * from src where key > 490").explain(true)
> sql("explain extended create table tb as select * from src where key > 490")



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message