spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Terry Kim <yumin...@gmail.com>
Subject [DISCUSS] Consistent relation resolution behavior in SparkSQL
Date Mon, 02 Dec 2019 01:12:04 GMT
Hi all,

As discussed in SPARK-29900, Spark currently has two different relation
resolution behaviors:

   1. Look up temp view first, then table/persistent view
   2. Look up table/persistent view

The first behavior is used in SELECT, INSERT and a few commands that
support temp views such as DESCRIBE TABLE, etc. The second behavior is used
in most commands. Thus, it is hard to predict which relation resolution
rule is being applied for a given command.

I want to propose a consistent relation resolution behavior in which temp
views are always looked up first before table/persistent view, as
described more in detail in this doc: consistent relation resolution
proposal
<https://docs.google.com/document/d/1hvLjGA8y_W_hhilpngXVub1Ebv8RsMap986nENCFnrg/edit?usp=sharing>
.

Note that this proposal is a breaking change, but the impact should be
minimal since this applies only when there are temp views and tables with
the same name.

Any feedback will be appreciated.

I also want to thank Wenchen Fan, Ryan Blue, Burak Yavuz, and Dongjoon Hyun
for guidance and suggestion.

Regards,
Terry


<https://issues.apache.org/jira/browse/SPARK-29900>

Mime
View raw message