flink-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (FLINK-5668) Reduce dependency on HDFS at job startup time
Date Fri, 24 Feb 2017 21:11:44 GMT

    [ https://issues.apache.org/jira/browse/FLINK-5668?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15883521#comment-15883521

ASF GitHub Bot commented on FLINK-5668:

GitHub user billliuatuber opened a pull request:


    [FLINK-5668] Reduce dependency on HDFS at job startup time

    In current implementation, Job manager writes task manager configuration into a file and
upload it to HDFS. This file's used to bootstrap taskmanager.
    In this PR, it switches to use system environment instead of HDFS file to pass the configuration
from job manager to task manager, which reduce the dependency on HDFS. 

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/billliuatuber/flink FLINK-5668

Alternatively you can review and apply these changes as the patch at:


To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #3413
commit 547b333203db8e01737a690e06ad5f5663d7faca
Author: Bill Liu <bill@uber.com>
Date:   2017-02-21T03:27:46Z

    [FLINK-5668] reduce hdfs dependency at startup time


> Reduce dependency on HDFS at job startup time
> ---------------------------------------------
>                 Key: FLINK-5668
>                 URL: https://issues.apache.org/jira/browse/FLINK-5668
>             Project: Flink
>          Issue Type: Improvement
>          Components: YARN
>            Reporter: Bill Liu
>   Original Estimate: 48h
>  Remaining Estimate: 48h
> When create a Flink cluster on Yarn,  JobManager depends on  HDFS to share  taskmanager-conf.yaml
 with TaskManager.
> It's better to share the taskmanager-conf.yaml  on JobManager Web server instead of HDFS,
which could reduce the HDFS dependency  at job startup.

This message was sent by Atlassian JIRA

View raw message