spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Wenchen Fan (JIRA)" <>
Subject [jira] [Resolved] (SPARK-25127) DataSourceV2: Remove SupportsPushDownCatalystFilters
Date Thu, 23 Aug 2018 00:12:00 GMT


Wenchen Fan resolved SPARK-25127.
       Resolution: Fixed
    Fix Version/s: 2.4.0

Issue resolved by pull request 22185

> DataSourceV2: Remove SupportsPushDownCatalystFilters
> ----------------------------------------------------
>                 Key: SPARK-25127
>                 URL:
>             Project: Spark
>          Issue Type: Sub-task
>          Components: SQL
>    Affects Versions: 2.3.0, 2.3.1, 2.4.0
>            Reporter: Ryan Blue
>            Assignee: Reynold Xin
>            Priority: Major
>             Fix For: 2.4.0
> Discussion about adding TableCatalog on the dev list focused around whether Expression
should be used in the public DataSourceV2 API, with SupportsPushDownCatalystFilters as an
example of where it is already exposed. The early consensus is that Expression should not
be exposed in the public API.
> From [~rxin]:
> bq. I completely disagree with using Expression in critical public APIs that we expect
a lot of developers to use . . . If we are depending on Expressions on the more common APIs
in dsv2 already, we should revisit that.
> The main use of this API is to pass Expression to FileFormat classes that used Expression
instead of Filter. External sources also use it for more complex push-down, like {{to_date(ts)
= '2018-05-13'}}, but those uses can be done with Analyzer rules or when translating to Filters.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message