spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Xiao Li (JIRA)" <>
Subject [jira] [Commented] (SPARK-12850) Support bucket pruning (predicate pushdown for bucketed tables)
Date Sun, 24 Jan 2016 18:10:39 GMT


Xiao Li commented on SPARK-12850:

Doing the design and prototype. Notice a few issues we need to consider:
- Partitioning columns could have overlapping columns with the bucketing columns;
- The predicates we can use for bucket pruning: EqualTo, EqualNullSafe, IsNull, In, InSet;
- Need to support mixed And and Or in the filters;
- After generating the bucket set we need to scan, we should remove the corresponding filters,
if possible;

Maybe, I will just submit a simplified version at first. 

> Support bucket pruning (predicate pushdown for bucketed tables)
> ---------------------------------------------------------------
>                 Key: SPARK-12850
>                 URL:
>             Project: Spark
>          Issue Type: Sub-task
>          Components: SQL
>            Reporter: Reynold Xin
> We now support bucketing. One optimization opportunity is to push some predicates into
the scan to skip scanning files that definitely won't match the values.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message