flink-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Zili Chen (Jira)" <j...@apache.org>
Subject [jira] [Commented] (FLINK-13938) Use yarn public distributed cache to speed up containers launch
Date Tue, 12 Nov 2019 07:00:00 GMT

    [ https://issues.apache.org/jira/browse/FLINK-13938?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16972117#comment-16972117

Zili Chen commented on FLINK-13938:

We also have a draft [~fly_in_gis]. I think we can merge two implementation if there is diff,
no matter base on which one. I am looking forward to your PR; otherwise I'm able to send a
PR and invite you to review it.

> Use yarn public distributed cache to speed up containers launch
> ---------------------------------------------------------------
>                 Key: FLINK-13938
>                 URL: https://issues.apache.org/jira/browse/FLINK-13938
>             Project: Flink
>          Issue Type: New Feature
>          Components: Deployment / YARN
>            Reporter: Yang Wang
>            Assignee: Yang Wang
>            Priority: Major
> By default, the LocalResourceVisibility is APPLICATION, so they will be downloaded only
once and shared for all taskmanager containers of a same application in the same node. However,
different applications will have to download all jars every time, including the flink-dist.jar.
I think we could use the yarn public cache to eliminate the unnecessary jars downloading and
make launching container faster.
> How to use the shared lib feature?
>  # Upload a copy of flink release binary to hdfs.
>  # Use the -ysl argument to specify the shared lib
> {code:java}
> ./bin/flink run -d -m yarn-cluster -p 20 -ysl hdfs:///flink/release/flink-1.9.0/lib examples/streaming/WindowJoin.jar{code}
> -ysl, --yarnsharedLib <path>          Upload a copy of flink lib beforehand
>                                                           and
specify the path to use public
>                                                           visibility
feature of YARN NodeManager
localizing resources.

This message was sent by Atlassian Jira

View raw message