flink-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (FLINK-3665) Range partitioning lacks support to define sort orders
Date Mon, 18 Apr 2016 15:01:25 GMT

    [ https://issues.apache.org/jira/browse/FLINK-3665?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15245815#comment-15245815
] 

ASF GitHub Bot commented on FLINK-3665:
---------------------------------------

Github user fhueske commented on a diff in the pull request:

    https://github.com/apache/flink/pull/1848#discussion_r60074102
  
    --- Diff: flink-optimizer/src/main/java/org/apache/flink/optimizer/dag/PartitionNode.java
---
    @@ -90,13 +90,20 @@ public SemanticProperties getSemanticProperties() {
     		private final PartitionMethod pMethod;
     		private final Partitioner<?> customPartitioner;
     		private final DataDistribution distribution;
    -		
    +		private final Ordering ordering;
    +
     		public PartitionDescriptor(PartitionMethod pMethod, FieldSet pKeys, Partitioner<?>
customPartitioner, DataDistribution distribution) {
    +			this(pMethod, pKeys, null, customPartitioner, distribution);
    +		}
    +
    +		public PartitionDescriptor(PartitionMethod pMethod, FieldSet pKeys, Ordering ordering,
Partitioner<?>
    +				customPartitioner, DataDistribution distribution) {
     			super(pKeys);
    -			
    +
     			this.pMethod = pMethod;
     			this.customPartitioner = customPartitioner;
     			this.distribution = distribution;
    +			this.ordering = ordering;
    --- End diff --
    
    Can you add a check that the `ordering` is valid (number of orders == number of keys.
Order keys == keys)?


> Range partitioning lacks support to define sort orders
> ------------------------------------------------------
>
>                 Key: FLINK-3665
>                 URL: https://issues.apache.org/jira/browse/FLINK-3665
>             Project: Flink
>          Issue Type: Improvement
>          Components: DataSet API
>    Affects Versions: 1.0.0
>            Reporter: Fabian Hueske
>             Fix For: 1.1.0
>
>
> {{DataSet.partitionByRange()}} does not allow to specify the sort order of fields. This
is fine if range partitioning is used to reduce skewed partitioning. 
> However, it is not sufficient if range partitioning is used to sort a data set in parallel.

> Since {{DataSet.partitionByRange()}} is {{@Public}} API and cannot be easily changed,
I propose to add a method {{withOrders(Order... orders)}} to {{PartitionOperator}}. The method
should throw an exception if the partitioning method of {{PartitionOperator}} is not range
partitioning.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message