flink-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (FLINK-7) [GitHub] Enable Range Partitioner
Date Wed, 04 Nov 2015 22:58:27 GMT

    [ https://issues.apache.org/jira/browse/FLINK-7?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14990640#comment-14990640
] 

ASF GitHub Bot commented on FLINK-7:
------------------------------------

Github user fhueske commented on a diff in the pull request:

    https://github.com/apache/flink/pull/1255#discussion_r43953382
  
    --- Diff: flink-runtime/src/main/java/org/apache/flink/runtime/operators/shipping/OutputEmitter.java
---
    @@ -67,27 +67,11 @@ public OutputEmitter(ShipStrategyType strategy) {
     	 * @param comparator The comparator used to hash / compare the records.
     	 */
     	public OutputEmitter(ShipStrategyType strategy, TypeComparator<T> comparator)
{
    -		this(strategy, comparator, null, null);
    +		this(strategy, comparator, null);
     	}
     	
    -	/**
    -	 * Creates a new channel selector that uses the given strategy (broadcasting, partitioning,
...)
    -	 * and uses the supplied comparator to hash / compare records for partitioning them
deterministically.
    -	 * 
    -	 * @param strategy The distribution strategy to be used.
    -	 * @param comparator The comparator used to hash / compare the records.
    -	 * @param distr The distribution pattern used in the case of a range partitioning.
    -	 */
    -	public OutputEmitter(ShipStrategyType strategy, TypeComparator<T> comparator,
DataDistribution distr) {
    -		this(strategy, comparator, null, distr);
    -	}
    -	
    -	public OutputEmitter(ShipStrategyType strategy, TypeComparator<T> comparator,
Partitioner<?> partitioner) {
    -		this(strategy, comparator, partitioner, null);
    -	}
    -		
     	@SuppressWarnings("unchecked")
    -	public OutputEmitter(ShipStrategyType strategy, TypeComparator<T> comparator,
Partitioner<?> partitioner, DataDistribution distr) {
    +	public OutputEmitter(ShipStrategyType strategy, TypeComparator<T> comparator,
Partitioner<?> partitioner) {
    --- End diff --
    
    I would keep the `DataDistribution` in the `OutputEmitter`. 
    We can later adapt this such that we can set the `DataDistribution` at execution time
from the broadcasted set and avoid the partition ID assignment.
    
    For now, we can simply set an Integer distribution with boundaries 1,2,3...,n. 


> [GitHub] Enable Range Partitioner
> ---------------------------------
>
>                 Key: FLINK-7
>                 URL: https://issues.apache.org/jira/browse/FLINK-7
>             Project: Flink
>          Issue Type: Sub-task
>          Components: Distributed Runtime
>            Reporter: GitHub Import
>            Assignee: Chengxiang Li
>             Fix For: pre-apache
>
>
> The range partitioner is currently disabled. We need to implement the following aspects:
> 1) Distribution information, if available, must be propagated back together with the
ordering property.
> 2) A generic bucket lookup structure (currently specific to PactRecord).
> Tests to re-enable after fixing this issue:
>  - TeraSortITCase
>  - GlobalSortingITCase
>  - GlobalSortingMixedOrderITCase
> ---------------- Imported from GitHub ----------------
> Url: https://github.com/stratosphere/stratosphere/issues/7
> Created by: [StephanEwen|https://github.com/StephanEwen]
> Labels: core, enhancement, optimizer, 
> Milestone: Release 0.4
> Assignee: [fhueske|https://github.com/fhueske]
> Created at: Fri Apr 26 13:48:24 CEST 2013
> State: open



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message