flink-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Nico Kruber (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (FLINK-6949) Add ability to ship custom resource files to YARN cluster
Date Fri, 14 Jul 2017 13:14:00 GMT

    [ https://issues.apache.org/jira/browse/FLINK-6949?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16087294#comment-16087294
] 

Nico Kruber commented on FLINK-6949:
------------------------------------

referring to the discussion we had on the user mailing list at https://lists.apache.org/thread.html/76eb9f2e30174b4cdb9b665682206a8aded52be606f89f3978db6444@%3Cuser.flink.apache.org%3E
:

A workaround is to use Flinks {{DistributedCache}}:
You can parse a job's parameter with your file URL and then give this to the distributed cache:

{code:java}
public static void main(String[] args) throws Exception {
        ParameterTool params = ParameterTool.fromArgs(args);

        ...

        env.registerCachedFile(params.get("config_file", <default/path>), 
<file_identifier>);

        ...
}
{code}

Flinks {{DistributedCache}} will then cache the file locally (on the task managers) and you
can use it in a RichFunction like in
https://github.com/apache/flink/blob/master/flink-tests/src/test/java/org/apache/flink/test/distributedCache/DistributedCacheTest.java#L99

{code:java}
public class MyFunction extends AbstractRichFunction {
        private static final long serialVersionUID = 1L;

        @Override
        public void open(Configuration conf) throws IOException {
                File file = 
getRuntimeContext().getDistributedCache().getFile(<file_identifier>);
...
        }
}
{code}

> Add ability to ship custom resource files to YARN cluster
> ---------------------------------------------------------
>
>                 Key: FLINK-6949
>                 URL: https://issues.apache.org/jira/browse/FLINK-6949
>             Project: Flink
>          Issue Type: Improvement
>          Components: Client, YARN
>    Affects Versions: 1.3.0
>            Reporter: Mikhail Pryakhin
>            Priority: Critical
>
> *The problem:*
> When deploying a flink job on YARN it is not possible to specify custom resource files
to be shipped to YARN cluster.
>  
> *The use case description:*
> When running a flink job on multiple environments it becomes necessary to pass environment-related
configuration files to the job's runtime. It can be accomplished by packaging configuration
files within the job's jar. But having tens of different environments one can easily end up
packaging as many jars as there are environments. It would be great to have an ability to
separate configuration files from the job artifacts. 
>  
> *The possible solution:*
> add the --yarnship-files option to flink cli to specify files that should be shipped
to the YARN cluster.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Mime
View raw message