flink-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (FLINK-5955) Merging a list of buffered records will have problem when ObjectReuse is turned on
Date Fri, 03 Mar 2017 05:59:45 GMT

    [ https://issues.apache.org/jira/browse/FLINK-5955?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15893780#comment-15893780
] 

ASF GitHub Bot commented on FLINK-5955:
---------------------------------------

GitHub user shaoxuan-wang opened a pull request:

    https://github.com/apache/flink/pull/3464

    [FLINK-5955] [table] Merging a list of buffered records will have problem when ObjectReuse
is turned on

    This PR changes the dataSet AGG merge to pair-merge. 
    
    If we buffer the iterated records for group-merge, we will get wrong error when ObjectReuse
is turned on. Alternatively, we could deep-copy every record and buffer them for group-merge.
But I think that is expense in terms of memory and also CPU. We could later add group-merge
when needed (in the future we should add rules to select either pair-merge or group-merge,
but for now all built-in aggregates should work fine with pair-merge).
    
    Thanks for contributing to Apache Flink. Before you open your pull request, please take
the following check list into consideration.
    If your changes take all of the items into account, feel free to open your pull request.
For more information and/or questions please refer to the [How To Contribute guide](http://flink.apache.org/how-to-contribute.html).
    In addition to going through the list, please provide a meaningful description of your
changes.
    
    - [X] General
      - The pull request references the related JIRA issue ("[FLINK-XXX] Jira title text")
      - The pull request addresses only one issue
      - Each commit in the PR has a meaningful commit message (including the JIRA id)
    
    - [ ] Documentation
      - Documentation has been added for new functionality
      - Old documentation affected by the pull request has been updated
      - JavaDoc for public methods has been added
    
    - [X] Tests & Build
      - Functionality added by the pull request is covered by tests
      - `mvn clean verify` has been executed successfully locally or a Travis build has passed


You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/shaoxuan-wang/flink F5955-submit

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/flink/pull/3464.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #3464
    
----
commit e6cdab7cd309f16d028894943f177f4321889630
Author: shaoxuan-wang <wshaoxuan@gmail.com>
Date:   2017-03-03T05:50:29Z

    [FLINK-5955] [table] Merging a list of buffered records will have problem when ObjectReuse
is turned on

----


> Merging a list of buffered records will have problem when ObjectReuse is turned on
> ----------------------------------------------------------------------------------
>
>                 Key: FLINK-5955
>                 URL: https://issues.apache.org/jira/browse/FLINK-5955
>             Project: Flink
>          Issue Type: Bug
>          Components: Table API & SQL
>            Reporter: Shaoxuan Wang
>            Assignee: Shaoxuan Wang
>
> Turn on ObjectReuse in MultipleProgramsTestBase:
> TestEnvironment clusterEnv = new TestEnvironment(cluster, 4, true);
> Then the tests "testEventTimeSessionGroupWindow", "testEventTimeSessionGroupWindow",
and "testEventTimeTumblingGroupWindowOverTime"  will fail.
> The reason is that we have buffered iterated records for group-merge. I think we should
change the Agg merge to pair-merge, and later add group-merge when needed (in the future we
should add rules to select either pair-merge or group-merge, but for now all built-in aggregates
should work fine with pair-merge).



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Mime
View raw message