spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ryan Blue (JIRA)" <>
Subject [jira] [Commented] (SPARK-24130) Data Source V2: Join Push Down
Date Sun, 13 May 2018 18:21:00 GMT


Ryan Blue commented on SPARK-24130:

[~jliwork] could you please open up access to allow open commenting on the design doc? I'd
like to add some questions and comments while I review it. Thank you!

> Data Source V2: Join Push Down
> ------------------------------
>                 Key: SPARK-24130
>                 URL:
>             Project: Spark
>          Issue Type: Sub-task
>          Components: SQL
>    Affects Versions: 2.3.0
>            Reporter: Jia Li
>            Priority: Major
> Spark applications often directly query external data sources such as relational databases,
or files. Spark provides Data Sources APIs for accessing structured data through Spark SQL.
Data Sources APIs in both V1 and V2 support optimizations such as Filter push down and Column
pruning which are subset of the functionality that can be pushed down to some data sources.
We’re proposing to extend Data Sources APIs with join push down (JPD). Join push down significantly
improves query performance by reducing the amount of data transfer and exploiting the capabilities
of the data sources such as index access.
> Join push down design document is available [here|].

This message was sent by Atlassian JIRA

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message