flink-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Danny Chen (Jira)" <j...@apache.org>
Subject [jira] [Commented] (FLINK-14716) Cooperate computed column with project push down
Date Tue, 12 Nov 2019 09:42:00 GMT

    [ https://issues.apache.org/jira/browse/FLINK-14716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16972228#comment-16972228

Danny Chen commented on FLINK-14716:

Thanks Jark for reporting this, i checked again the following rules:

* PushFilterIntoTableSourceScanRule // affected
* PushLimitIntoTableSourceScanRule // affected
* PushPartiitonIntoTableSourceScanRule
* PushProjectIntoTableSourceScanRule

The first 2 rules are affected by change FLINK-14665 if the source table is with computed
column. The last 2 are still correct.

The DatabaseCalciteSchema returns a TableSourceTable when we use getTable from Calcite catalog
reader, then this TableSourceTable was wrapped as a FlinkRelOptTable, this FlinkRelOptTable
is the role to do the computed column conversion. This is the reason why i change the TableSourceTable
row type to include the viarual columns.

In order to solve this, we may need to make sure what the TableSourceTable is(then we know
that row type is should return), i figure out 2 ways:
# If we make sure that TableSourceTable return type should return virtual columns, then fixing
the last 2 rules is okey.
# If we make sure that TableSourceTable return type should not include virtual columns, then
we may need another "Table" abstraction to return a with virtual columns table when invoking

> Cooperate computed column with project push down
> ------------------------------------------------
>                 Key: FLINK-14716
>                 URL: https://issues.apache.org/jira/browse/FLINK-14716
>             Project: Flink
>          Issue Type: Sub-task
>          Components: Table SQL / Planner
>            Reporter: Jark Wu
>            Priority: Major
> In FLINK-14665, we supported computed column in planner. However, the implementation
of {{TableSourceTable#getRowType}} will include computed columns which is possibly not right,
because computed column is generated in the downstream Project, not in the source scan. 
> This is possibly lead to errors when push project into source scan which will call {{TableSourceTable#getRowType}}
in the rule. Currently, all the tests are passed because we don't have a test to cover this,
i.e. computed column on a projectable TableSource. We should at least add a test to cover

This message was sent by Atlassian Jira

View raw message