spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Леонид Поляков <>
Subject Custom DataFrame filter
Date Fri, 17 Jun 2016 08:45:30 GMT
Hi all!

Spark 1.6.1.

Anyone know how to implement custom DF filter to be later pushed down to
custom datasource?

To be short, I've managed to create custom Expression, implicitly add
methods with it to Column class, but I am stuck at the point where
Expression must be converted to Filter by Spark before it is passed to
"unhandledFilters" method at datasource Relation.

Take a look at DataSourceStrategy.translateFilter - this is the place where
mapping between Expression and Filter is hardcoded into Spark.
Anyone know a good way to put just another custom filter in there?
Or anyone implemented a custom filtering before + pushdown to datasource?

Thanks in advance!

View raw message