spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Michael Armbrust <>
Subject Re: Spark SQL Custom Predicate Pushdown
Date Sat, 17 Jan 2015 20:38:00 GMT
> 1) The fields in the SELECT clause are not pushed down to the predicate
> pushdown API. I have many optimizations that allow fields to be filtered
> out before the resulting object is serialized on the Accumulo tablet
> server. How can I get the selection information from the execution plan?
> I'm a little hesitant to implement the data relation that allows me to see
> the logical plan because it's noted in the comments that it could change
> without warning.

I'm not sure I understand.  The list of required columns should be pushed
down to the data source.  Are you looking for something more complicated?

> 2) I'm surprised to find that the predicate pushdown filters get
> completely removed when I do anything more complex in a where clause other
> than simple AND statements. Using an OR statement caused the filter array
> that was passed into the PrunedFilteredDataSource to be empty.

This was just an initial cut at the set of predicates to push down.  We can
add Or.  Mind opening a JIRA?

View raw message