flink-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Robert Metzger (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (FLINK-5668) Reduce dependency on HDFS at job startup time
Date Mon, 27 Feb 2017 09:45:45 GMT

    [ https://issues.apache.org/jira/browse/FLINK-5668?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15885456#comment-15885456
] 

Robert Metzger commented on FLINK-5668:
---------------------------------------

Sorry that I did not look at this JIRA earlier.

[~bill.liu8904] and [~wheat9] if I understand you correctly, you want Flink on YARN not to
use Hadoop's {{fs.defaultFS}} configuration for choosing the filesystem used to distribute
jars and configuration files during deployment?

This basically means that we need to provide a custom configuration key in Flink (something
like {{yarn.deploy.fs}}) to put the stuff to. This would allow you so use s3 for deploying
Flink and hdfs for rocksdb or other state backups.

I'm not able to completely understand FLINK-5631: How can I register an additional jar from
a different file system as a required resource?

> Reduce dependency on HDFS at job startup time
> ---------------------------------------------
>
>                 Key: FLINK-5668
>                 URL: https://issues.apache.org/jira/browse/FLINK-5668
>             Project: Flink
>          Issue Type: Improvement
>          Components: YARN
>            Reporter: Bill Liu
>   Original Estimate: 48h
>  Remaining Estimate: 48h
>
> When create a Flink cluster on Yarn,  JobManager depends on  HDFS to share  taskmanager-conf.yaml
 with TaskManager.
> It's better to share the taskmanager-conf.yaml  on JobManager Web server instead of HDFS,
which could reduce the HDFS dependency  at job startup.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Mime
View raw message