flink-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Fabian Hueske (JIRA)" <j...@apache.org>
Subject [jira] [Created] (FLINK-7696) Add projection push-down support for TableSources with time attributes
Date Tue, 26 Sep 2017 17:36:00 GMT
Fabian Hueske created FLINK-7696:

             Summary: Add projection push-down support for TableSources with time attributes
                 Key: FLINK-7696
                 URL: https://issues.apache.org/jira/browse/FLINK-7696
             Project: Flink
          Issue Type: Improvement
          Components: Table API & SQL
    Affects Versions: 1.4.0
            Reporter: Fabian Hueske

Table sources that implement the {{DefinedProctimeAttribute}} or {{DefinedRowtimeAttribute}}
do not support projection push-down even if they also implement {{ProjectableTableSource}}.

There are several problems:
- the schema of a {{TableSource}} that implements {{DefinedRowtimeAttribute}} or {{DefinedProctimeAttribute}}
is constructed in the catalog not in the {{TableSource}} (proctime fields are always appended
at the end).
- the {{ProjectableTableSource.projectFields()}} method returns the projected fields as int
indicies. In order to handle the indicies correctly, the TableSource would need to know about
the internals of the Table API.
- {{ProjectableTableSource.projectFields()}} might reorder fields and move a proctime field
into the middle of the schema. However, the TableSource has no control over that.
- A {{TableSource}} that implements {{DefinedRowtimeAttribute}} or {{DefinedProctimeAttribute}}
would need to change the return values of {{getRowtimeAttribute()}} or {{getProctimeAttribute()}}
depending on whether the attribute is kept or not.

Adjusting the schema of table sources inside of the Table API makes all of this quite messy.
Maybe we need to refine the interfaces. For instance, we could ask users to explicitly add
time indicator fields in the {{TypeInformation}} returned by {{TableSource.getReturnType()}}.
However, that might collide with plans to add computable time attributes as proposed in FLINK-7548.

This message was sent by Atlassian JIRA

View raw message