spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Wenchen Fan <>
Subject Re: [DISCUSS] Consistent relation resolution behavior in SparkSQL
Date Thu, 05 Dec 2019 03:02:21 GMT
+1, I think it's good for both end-users and Spark developers:
* for end-users, when they lookup a table, they don't need to care which
command triggers it, as the behavior is consistent in all the places.
* for Spark developers, we may simplify the code quite a bit. For now we
have two code paths to lookup tables: one for SELECT/INSERT and one for
other commands.


On Mon, Dec 2, 2019 at 9:12 AM Terry Kim <> wrote:

> Hi all,
> As discussed in SPARK-29900, Spark currently has two different relation
> resolution behaviors:
>    1. Look up temp view first, then table/persistent view
>    2. Look up table/persistent view
> The first behavior is used in SELECT, INSERT and a few commands that
> support temp views such as DESCRIBE TABLE, etc. The second behavior is used
> in most commands. Thus, it is hard to predict which relation resolution
> rule is being applied for a given command.
> I want to propose a consistent relation resolution behavior in which temp
> views are always looked up first before table/persistent view, as
> described more in detail in this doc: consistent relation resolution
> proposal
> <>
> .
> Note that this proposal is a breaking change, but the impact should be
> minimal since this applies only when there are temp views and tables with
> the same name.
> Any feedback will be appreciated.
> I also want to thank Wenchen Fan, Ryan Blue, Burak Yavuz, and Dongjoon
> Hyun for guidance and suggestion.
> Regards,
> Terry
> <>

View raw message