flink-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Haohui Mai (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (FLINK-5668) Reduce dependency on HDFS at job startup time
Date Mon, 27 Feb 2017 19:12:45 GMT

    [ https://issues.apache.org/jira/browse/FLINK-5668?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15886342#comment-15886342

Haohui Mai commented on FLINK-5668:

[~rmetzger] -- just want to clarify FLINK-5631 here.

YARN downloads the resources from the specified paths and localizes the resources on worker
nodes. Note that the {{Path}} class in the Hadoop APIs supports specifying filesystem other
than the one specified in {{default.FS}}. For example, {{new Path(URI.create("s3a://foo")}}
specifies the a resource on S3, regardless what {{default.FS}} is specified. FLINK-5631 enables
YARN to localize resources that are not stored on {{default.FS}}.

> Reduce dependency on HDFS at job startup time
> ---------------------------------------------
>                 Key: FLINK-5668
>                 URL: https://issues.apache.org/jira/browse/FLINK-5668
>             Project: Flink
>          Issue Type: Improvement
>          Components: YARN
>            Reporter: Bill Liu
>   Original Estimate: 48h
>  Remaining Estimate: 48h
> When create a Flink cluster on Yarn,  JobManager depends on  HDFS to share  taskmanager-conf.yaml
 with TaskManager.
> It's better to share the taskmanager-conf.yaml  on JobManager Web server instead of HDFS,
which could reduce the HDFS dependency  at job startup.

This message was sent by Atlassian JIRA

View raw message