spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Apache Spark (JIRA)" <>
Subject [jira] [Commented] (SPARK-20331) Broaden support for Hive partition pruning predicate pushdown
Date Sat, 21 Oct 2017 07:11:00 GMT


Apache Spark commented on SPARK-20331:

User 'gatorsmile' has created a pull request for this issue:

> Broaden support for Hive partition pruning predicate pushdown
> -------------------------------------------------------------
>                 Key: SPARK-20331
>                 URL:
>             Project: Spark
>          Issue Type: Improvement
>          Components: SQL
>    Affects Versions: 2.1.0
>            Reporter: Michael Allman
>            Assignee: Michael Allman
>             Fix For: 2.3.0
> Spark 2.1 introduced scalable support for Hive tables with huge numbers of partitions.
Key to leveraging this support is the ability to prune unnecessary table partitions to answer
queries. Spark supports a subset of the class of partition pruning predicates that the Hive
metastore supports. If a user writes a query with a partition pruning predicate that is *not*
supported by Spark, Spark falls back to loading all partitions and pruning client-side. We
want to broaden Spark's current partition pruning predicate pushdown capabilities.
> One of the key missing capabilities is support for disjunctions. For example, for a table
partitioned by date, specifying with a predicate like
> {code}date = 20161011 or date = 20161014{code}
> will result in Spark fetching all partitions. For a table partitioned by date and hour,
querying a range of hours across dates can be quite difficult to accomplish without fetching
all partition metadata.
> The current partition pruning support supports only comparisons against literals. We
can expand that to foldable expressions by evaluating them at planning time.
> We can also implement support for the "IN" comparison by expanding it to a sequence of
> This ticket covers those enhancements.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message