flink-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (FLINK-2997) Support range partition with user customized data distribution.
Date Wed, 23 Mar 2016 11:05:25 GMT

    [ https://issues.apache.org/jira/browse/FLINK-2997?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15208234#comment-15208234
] 

ASF GitHub Bot commented on FLINK-2997:
---------------------------------------

Github user fhueske commented on a diff in the pull request:

    https://github.com/apache/flink/pull/1776#discussion_r57141500
  
    --- Diff: flink-tests/src/test/java/org/apache/flink/test/javaApiOperators/CustomDistributionITCase.java
---
    @@ -94,12 +104,10 @@ public void testRangeWithDistribution2() throws Exception{
     			@Override
     			public void mapPartition(Iterable<Tuple3<Integer, Integer, String>> values,
Collector<Boolean> out) throws Exception {
     				int partitionIndex = getRuntimeContext().getIndexOfThisSubtask();
    -
     				for (Tuple3<Integer, Integer, String> s : values) {
    -					if (s.f0 <= partitionIndex * (partitionIndex + 1) / 2 ||
    -							s.f0 > (partitionIndex + 1) * (partitionIndex + 2) / 2 ||
    -							s.f1 - 1 != partitionIndex) {
    -						fail("Record was not correctly partitioned: " + s.toString());
    +					
    +					if ((s.f0 > partitionIndex + 1) || ((s.f0 == partitionIndex + 1) && (s.f1
> dist.rightBoundary[partitionIndex]))) {
    --- End diff --
    
    The test checks only one bucket boundary. We need to check 
    - the upper boundary if `partitionIndex == 0`
    - lower and upper boundaries if `0 < `partitionIndex` < 4`
    - the lower boundary if `partitionIndex == 4.


> Support range partition with user customized data distribution.
> ---------------------------------------------------------------
>
>                 Key: FLINK-2997
>                 URL: https://issues.apache.org/jira/browse/FLINK-2997
>             Project: Flink
>          Issue Type: New Feature
>            Reporter: Chengxiang Li
>
> This is a followup work of FLINK-7, sometime user have better knowledge of the source
data, and they can build customized data distribution to do range partition more efficiently.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message