spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jacek Laskowski (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (SPARK-20122) Analyzer reports "unresolved operator 'Aggregate" for nonexistent window specifications
Date Tue, 28 Mar 2017 06:37:41 GMT

    [ https://issues.apache.org/jira/browse/SPARK-20122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15944631#comment-15944631
] 

Jacek Laskowski commented on SPARK-20122:
-----------------------------------------

Once agreed it's an issue, I'd appreciate any guidance how to solve it and would like to seat
on it. Thanks.

> Analyzer reports "unresolved operator 'Aggregate" for nonexistent window specifications
> ---------------------------------------------------------------------------------------
>
>                 Key: SPARK-20122
>                 URL: https://issues.apache.org/jira/browse/SPARK-20122
>             Project: Spark
>          Issue Type: Improvement
>          Components: SQL
>    Affects Versions: 2.2.0
>            Reporter: Jacek Laskowski
>            Priority: Minor
>
> {{Analyzer}} should be more elaborate on the root cause of the {{org.apache.spark.sql.AnalysisException}}
when {{WindowSpecReference}} cannot be resolved.
> I think the real issue is that {{AstBuilder}} creates a lookup table for {{WindowSpecDefinition}}s
per name and when it cannot resolve {{WindowSpecReference}} it simply fails analysis (it's
too early in the SQL parsing).
> {code}
> scala> val table = spark.emptyDataset[Int]
> table: org.apache.spark.sql.Dataset[Int] = [value: int]
> scala> table.selectExpr("count() OVER nonexistentWindowSpec")
> org.apache.spark.sql.AnalysisException: unresolved operator 'Aggregate [unresolvedwindowexpression(count(),
WindowSpecReference(nonexistentWindowSpec)) AS unresolvedwindowexpression(count())#15];;
> 'Aggregate [unresolvedwindowexpression(count(), WindowSpecReference(nonexistentWindowSpec))
AS unresolvedwindowexpression(count())#15]
> +- LocalRelation <empty>, [value#11]
>   at org.apache.spark.sql.catalyst.analysis.CheckAnalysis$class.failAnalysis(CheckAnalysis.scala:40)
>   at org.apache.spark.sql.catalyst.analysis.Analyzer.failAnalysis(Analyzer.scala:90)
>   at org.apache.spark.sql.catalyst.analysis.CheckAnalysis$$anonfun$checkAnalysis$3.apply(CheckAnalysis.scala:408)
>   at org.apache.spark.sql.catalyst.analysis.CheckAnalysis$$anonfun$checkAnalysis$3.apply(CheckAnalysis.scala:407)
>   at org.apache.spark.sql.catalyst.trees.TreeNode.foreachUp(TreeNode.scala:127)
>   at org.apache.spark.sql.catalyst.analysis.CheckAnalysis$class.checkAnalysis(CheckAnalysis.scala:407)
>   at org.apache.spark.sql.catalyst.analysis.Analyzer.checkAnalysis(Analyzer.scala:90)
>   at org.apache.spark.sql.execution.QueryExecution.assertAnalyzed(QueryExecution.scala:53)
>   at org.apache.spark.sql.Dataset$.ofRows(Dataset.scala:67)
>   at org.apache.spark.sql.Dataset.org$apache$spark$sql$Dataset$$withPlan(Dataset.scala:2832)
>   at org.apache.spark.sql.Dataset.select(Dataset.scala:1137)
>   at org.apache.spark.sql.Dataset.selectExpr(Dataset.scala:1172)
>   ... 48 elided
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message