flink-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Vasia Kalavri (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (FLINK-3888) Custom Aggregator with Convergence can't be registered directly with DeltaIteration
Date Mon, 03 Oct 2016 18:59:20 GMT

    [ https://issues.apache.org/jira/browse/FLINK-3888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15543133#comment-15543133
] 

Vasia Kalavri commented on FLINK-3888:
--------------------------------------

I had a quick look into this and I don't see any fundamental reason why we can't add a custom
convergence criterion in delta iterations. It seems that it is currently allowed to add one
convergence criterion only, and since delta iterations have the {{WorksetEmptyConvergenceCriterion}}
by default, adding a custom one is not possible.

So, one solution could be the following:
1. use {{TaskConfig.setConvergenceCriterion()}} to set the custom, user-defined convergence
criterion (like in the case of bulk iteration)
2. add a new method {{TaskConfig.setDefaultConvergeCriterion()}} to add the default empty
workset convergence
​3. check both criteria in {{IterationSynchronizationSinkTask.checkForConvergence()​}}
4. expose the custom convergence criterion in {{DeltaIteration}}

If I'm not missing something and this seems acceptable I'd like to resolve this issue. Custom
convergence would be helpful in several Gelly algorithms.

> Custom Aggregator with Convergence can't be registered directly with DeltaIteration
> -----------------------------------------------------------------------------------
>
>                 Key: FLINK-3888
>                 URL: https://issues.apache.org/jira/browse/FLINK-3888
>             Project: Flink
>          Issue Type: Bug
>          Components: Iterations
>            Reporter: Martin Liesenberg
>
> Contrary to the [documentation|https://ci.apache.org/projects/flink/flink-docs-master/apis/batch/iterations.html]
the method to add an aggregator with a custom convergence criterion to a DeltaIteration is
not exposed directly to DeltaIteration, but can only be accessed via the {{aggregatorRegistry}}.
> Moreover, when registering an aggregator with a custom convergence criterion  and running
the program, the following exception appears in the logs:
> {noformat}
> Error: Cannot use custom convergence criterion with workset iteration. Workset iterations
have implicit convergence criterion where workset is empty.
> org.apache.flink.optimizer.CompilerException: Error: Cannot use custom convergence criterion
with workset iteration. Workset iterations have implicit convergence criterion where workset
is empty.
> 	at org.apache.flink.optimizer.plantranslate.JobGraphGenerator.finalizeWorksetIteration(JobGraphGenerator.java:1518)
> 	at org.apache.flink.optimizer.plantranslate.JobGraphGenerator.compileJobGraph(JobGraphGenerator.java:198)
> 	at org.apache.flink.optimizer.plantranslate.JobGraphGenerator.compileJobGraph(JobGraphGenerator.java:164)
> 	at org.apache.flink.test.util.TestEnvironment.execute(TestEnvironment.java:76)
> 	at org.apache.flink.api.java.ExecutionEnvironment.execute(ExecutionEnvironment.java:898)
> 	at org.apache.flink.api.java.DataSet.collect(DataSet.java:410)
> 	at org.apache.flink.api.java.DataSet.print(DataSet.java:1605)
> {noformat}
> The issue has been found while discussing FLINK-2926



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message