spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Corey Nolet <>
Subject Re: Spark SQL Custom Predicate Pushdown
Date Sat, 17 Jan 2015 20:54:42 GMT

What I'm seeing (in Spark 1.2.0) is that the required columns being pushed
down to the DataRelation are not the product of the SELECT clause but
rather just the columns explicitly included in the WHERE clause.

Examples from my testing:

SELECT * FROM myTable --> The required columns are empty.
SELECT key1 FROM myTable --> The required columns are empty
SELECT * FROM myTable where key1 = 'val1' --> The required columns contains
SELECT key1,key2 FROM myTable where key1 = 'val1' --> The required columns
contains key1
SELECT key1,key2 FROM myTable where key1 = 'val1' and key2 = 'val2' --> The
required columns cintains key1,key2

I created SPARK-5296 for the OR predicate to be pushed down in some

On Sat, Jan 17, 2015 at 3:38 PM, Michael Armbrust <>

> 1) The fields in the SELECT clause are not pushed down to the predicate
>> pushdown API. I have many optimizations that allow fields to be filtered
>> out before the resulting object is serialized on the Accumulo tablet
>> server. How can I get the selection information from the execution plan?
>> I'm a little hesitant to implement the data relation that allows me to see
>> the logical plan because it's noted in the comments that it could change
>> without warning.
> I'm not sure I understand.  The list of required columns should be pushed
> down to the data source.  Are you looking for something more complicated?
>> 2) I'm surprised to find that the predicate pushdown filters get
>> completely removed when I do anything more complex in a where clause other
>> than simple AND statements. Using an OR statement caused the filter array
>> that was passed into the PrunedFilteredDataSource to be empty.
> This was just an initial cut at the set of predicates to push down.  We
> can add Or.  Mind opening a JIRA?

View raw message