flink-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Fabian Hueske (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (FLINK-2942) Dangling operators in web UI's program visualization (non-deterministic)
Date Mon, 09 Nov 2015 13:50:11 GMT

    [ https://issues.apache.org/jira/browse/FLINK-2942?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14996547#comment-14996547
] 

Fabian Hueske commented on FLINK-2942:
--------------------------------------

OK, the program does not exactly produce the previous problems but also results in a broken
flow graph:
As the 4th screenshot shows, the second join does only have a single input. 
Due to the various problems with different jobs, can this be a general problem with the graph
layout algorithm of the plotting library?

{code}
public static void main(String[] args) throws Exception {

  ExecutionEnvironment env = ExecutionEnvironment.getExecutionEnvironment();

  DataSet<Long> longs = env.generateSequence(0, 10000);
  DataSet<Long> fLongs = longs
      .filter(
        new FilterFunction<Long>() {
          @Override
          public boolean filter(Long l) throws Exception {
            return l < 100;
          }
        });

  DataSet<Tuple3<Long,Long,Long>> result = longs
      .join(fLongs).where("*").equalTo("*")
      .join(fLongs).where("f0").equalTo("*").with(
        new JoinFunction<Tuple2<Long, Long>, Long, Tuple3<Long, Long, Long>>()
{
          @Override
          public Tuple3<Long, Long, Long> join(Tuple2<Long, Long> a, Long b) throws
Exception {
            return new Tuple3<Long,Long,Long>(a.f0, a.f1, b);
          }
        });

  result.print();
}
{code}

> Dangling operators in web UI's program visualization (non-deterministic)
> ------------------------------------------------------------------------
>
>                 Key: FLINK-2942
>                 URL: https://issues.apache.org/jira/browse/FLINK-2942
>             Project: Flink
>          Issue Type: Bug
>          Components: Webfrontend
>    Affects Versions: 0.10
>         Environment: OSX, Firefox and Chrome
>            Reporter: Fabian Hueske
>            Priority: Critical
>             Fix For: 0.10, 1.0
>
>         Attachments: Screen Shot 2015-10-29 at 17.11.19.png, Screen Shot 2015-10-29 at
20.51.46.png, Screen Shot 2015-10-29 at 20.52.13.png
>
>
> When visualizing a program with three {{MapPartition}} operators that branch off from
an {{OuterJoin}} operator, two of the three {{MapPartition}} operators are not connected to
the {{OuterJoin}} operator and appear to have no input.
> The problem is present in FireFox as well as in Chrome. I'll attach a screenshot.
> The problem and be reproduced by executing the "Cascading for the impatient" [TFIDF example
program|https://github.com/Cascading/Impatient/tree/master/part5] using the [Cascading Flink
Connector|https://github.com/dataArtisans/cascading-flink].
> Update: It appears that the problem is non-deterministic. I ran the same job again (same
setup) and the previously missing connections were visualized. However, the UI showed only
one input for a binary operator (OuterJoin). Running the job a third time resulted in a graph
layout which was again different from both runs before. However, two of the {{MapPartition}}
operators had not inputs just as in the first run.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message