flink-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jeff Zhang (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (FLINK-11049) Unable to execute partial DAG
Date Fri, 07 Dec 2018 14:59:00 GMT

    [ https://issues.apache.org/jira/browse/FLINK-11049?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16712947#comment-16712947
] 

Jeff Zhang commented on FLINK-11049:
------------------------------------

[~till.rohrmann] IIUC, there're 2 kinds of executions in flink, one is eager execution (such
as print, collect or count), the other is lazy execution (such as writeAsText and etc). Eager
execution method (print, collect or count) will trigger the execution at once, while the lazy
execution is triggered by Env#execute method. I believe user would expect eager execution
method to only trigger the eager execution job, while Env#execute will trigger all the lazy
executions. 
In the examples of ticket description, users would expect print method to only execute the
print job but not the writeAsText job which should be triggered by Env#execute method explicitly.
 

This is what I make changes in DataSet#collect which will only trigger the print job rather
than all the jobs
https://github.com/apache/flink/pull/7214/files#diff-252c0e8c429fa9d7bec11e4f64ea4f0bR413


> Unable to execute partial DAG
> -----------------------------
>
>                 Key: FLINK-11049
>                 URL: https://issues.apache.org/jira/browse/FLINK-11049
>             Project: Flink
>          Issue Type: Bug
>          Components: Job-Submission
>    Affects Versions: 1.7.0
>            Reporter: Jeff Zhang
>            Assignee: Jeff Zhang
>            Priority: Major
>              Labels: pull-request-available
>
> {code}
> val benv = ExecutionEnvironment.getExecutionEnvironment
> val btEnv = TableEnvironment.getTableEnvironment(benv)
> val data = benv.fromElements("hello world", "hello flink", "hello hadoop")
> data.writeAsText("/Users/jzhang/a.txt", FileSystem.WriteMode.OVERWRITE);
> val table = data.flatMap(line=>line.split("\\s")).
>   map(w => (w, 1)).
>   toTable(btEnv, 'word, 'number)
> btEnv.registerTable("wc", table)
> btEnv.sqlQuery("select word, count(1) from wc group by word").
>   toDataSet[Row].print()
> {code}
> In the above example, the last statement will trigger 2 job execution (writeAsText and
print), but what user expect is the print job. The root cause is that currently, flink unable
to submit partial dag. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message