flink-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (FLINK-4297) Yarn client can't determine fat jar location if path contains spaces
Date Wed, 03 Aug 2016 13:03:20 GMT

    [ https://issues.apache.org/jira/browse/FLINK-4297?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15405886#comment-15405886
] 

ASF GitHub Bot commented on FLINK-4297:
---------------------------------------

Github user StephanEwen commented on a diff in the pull request:

    https://github.com/apache/flink/pull/2320#discussion_r73333859
  
    --- Diff: flink-yarn/src/main/java/org/apache/flink/yarn/cli/FlinkYarnSessionCli.java
---
    @@ -260,8 +263,15 @@ public AbstractYarnClusterDescriptor createDescriptor(String defaultApplicationN
     		} else {
     			LOG.info("No path for the flink jar passed. Using the location of "
     				+ yarnClusterDescriptor.getClass() + " to locate the jar");
    -			localJarPath = new Path("file://" +
    -				yarnClusterDescriptor.getClass().getProtectionDomain().getCodeSource().getLocation().getPath());
    +			String encodedJarPath =
    +				yarnClusterDescriptor.getClass().getProtectionDomain().getCodeSource().getLocation().getPath();
    +			try {
    +				// we have to decode the url encoded parts of the path
    +				localJarPath = new Path("file://" + URLDecoder.decode(encodedJarPath, Charset.defaultCharset().name()));
    --- End diff --
    
    I think manually coding the `file:// + path` URIs have cross-platform issues.
    
    For File URIs, I found it to be best to go with 
    ```java
    new File(string).toURI()
    ```
    or in that specific case:
    ```java
    localJarPath = new Path(new File(URLDecoder.decode(encodedJarPath, Charset.defaultCharset().name())).toURI());
    or
    localJarPath = new Path(new URI("file", null, URLDecoder.decode(encodedJarPath, Charset.defaultCharset().name()),
null));
    ```


> Yarn client can't determine fat jar location if path contains spaces
> --------------------------------------------------------------------
>
>                 Key: FLINK-4297
>                 URL: https://issues.apache.org/jira/browse/FLINK-4297
>             Project: Flink
>          Issue Type: Bug
>          Components: YARN Client
>            Reporter: Maximilian Michels
>            Assignee: Maximilian Michels
>             Fix For: 1.1.0, 1.2.0
>
>
> The code that automatically determines the fat jar path through the ProtectionDomain
of the Yarn class, receives a possibly URL encoded path string. We need to decode using the
system locale encoding, otherwise we can receive errors of the following when spaces are in
the file path: 
> {noformat}
> Caused by: java.io.FileNotFoundException: File file:/Users/max/Downloads/release-testing/flink-1.1.0-rc1/flink-1.1.0/build%20target/lib/flink-dist_2.11-1.1.0.jar
does not exist
>         at org.apache.hadoop.fs.RawLocalFileSystem.deprecatedGetFileStatus(RawLocalFileSystem.java:511)
>         at org.apache.hadoop.fs.RawLocalFileSystem.getFileLinkStatusInternal(RawLocalFileSystem.java:724)
>         at org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:501)
>         at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:397)
>         at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:337)
>         at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:289)
>         at org.apache.hadoop.fs.LocalFileSystem.copyFromLocalFile(LocalFileSystem.java:82)
>         at org.apache.hadoop.fs.FileSystem.copyFromLocalFile(FileSystem.java:1836)
>         at org.apache.flink.yarn.Utils.setupLocalResource(Utils.java:129)
>         at org.apache.flink.yarn.AbstractYarnClusterDescriptor.deployInternal(AbstractYarnClusterDescriptor.java:616)
>         at org.apache.flink.yarn.AbstractYarnClusterDescriptor.deploy(AbstractYarnClusterDescriptor.java:365)
>         ... 6 more
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message