flink-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Yang Wang (Jira)" <j...@apache.org>
Subject [jira] [Commented] (FLINK-14582) Do not upload {uuid}-taskmanager-conf.yaml for each task manager container
Date Thu, 31 Oct 2019 12:21:00 GMT

    [ https://issues.apache.org/jira/browse/FLINK-14582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16963942#comment-16963942
] 

Yang Wang commented on FLINK-14582:
-----------------------------------

>From the experience of our cluster, uploading a conf file to hdfs will take more than
50ms. If the name node of hdfs cluster is busy, it will task several seconds on a big hdfs
cluster. 

The time cost leads that we could only remove 10 container request in {{onContainersAllocated}}
when {{containerRequestHeartbeat}} is 500ms. So the {{YarnResourceManager}} will allocate
more containers than it really needs.

> Do not upload {uuid}-taskmanager-conf.yaml for each task manager container
> --------------------------------------------------------------------------
>
>                 Key: FLINK-14582
>                 URL: https://issues.apache.org/jira/browse/FLINK-14582
>             Project: Flink
>          Issue Type: Improvement
>          Components: Deployment / YARN
>    Affects Versions: 1.10.0, 1.9.1
>            Reporter: Yang Wang
>            Priority: Minor
>
> Currently when we {{createTaskExecutorLaunchContext}}, a new task manager conf will be
created and uploaded to hdfs. The time cost is unnecessary and could be reduced by using java
options of task manager main class instead.
> By reducing the time of launch container in {{onContainersAllocated}}, we could avoid
allocating excess containers as much as possible.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Mime
View raw message