spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Wenchen Fan (Jira)" <>
Subject [jira] [Resolved] (SPARK-29966) Add version method in TableCatalog to avoid load table twice
Date Fri, 06 Dec 2019 07:48:00 GMT


Wenchen Fan resolved SPARK-29966.
    Fix Version/s: 3.0.0
         Assignee: Terry Kim
       Resolution: Fixed

> Add version method in TableCatalog to avoid load table twice
> ------------------------------------------------------------
>                 Key: SPARK-29966
>                 URL:
>             Project: Spark
>          Issue Type: Improvement
>          Components: SQL
>    Affects Versions: 3.0.0
>            Reporter: ulysses you
>            Assignee: Terry Kim
>            Priority: Minor
>             Fix For: 3.0.0
> Now resolve logic plan will load table twice which are in ResolveTables and ResolveRelations.
The ResolveRelations is old code path, and ResolveTables is v2 code path, and the reason why
load table twice is that ResolveTables will load table and rollback v1 table to ResolveRelations
code path.
> The same scene also exists in ResolveSessionCatalog.
> It affect that execute command will cost double time than spark 2.4.
> Here is the idea that add a table version method in TableCatalog, and rules should always
get table version firstly without load table.

This message was sent by Atlassian Jira

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message