hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sanjay Radia (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-9357) Fallback to default authority if not specified in FileContext
Date Tue, 02 Apr 2013 01:21:16 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-9357?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13619403#comment-13619403
] 

Sanjay Radia commented on HADOOP-9357:
--------------------------------------

Failing when authority-port is not specified was consciously done when fileContext was implemented.
* A user wanting /tmp on the default file system should say "/tmp" and not "hdfs:///tmp".
* While the notion of a default file system is very appropriate, the notion of default authority
per scheme does not make sense, especially if it is derived from the default file system.

* In a multi-cluster environment, getting in the default authority/port from the default file
system can lead to accessing the files at the wrong HDFS cluster/volume (each cluster has
its own default and a user is allowed to access the HDFS volumes in different HDFS clusters
from within the same application (as in distcp)).
* If you encourage the use of hdfs:///tmp then as you change your default file system from
hdfs to localfs, the app can suddenly fail.  Being able to change the local file system to
different HDFS volumes/clusters, s3, localfs in a fairly transparent fashion is a useful feature
of Hadoop. For example, if /tmp  refers to the tmp on the default file system then things
work transparently when the the default file system is changed from hdfs://cluster1NN:37 to
hdfs://cluster2NN:55 to file:/// to s3fs etc. 

Is this Jira being driven by customer apps that are breaking?
                
> Fallback to default authority if not specified in FileContext
> -------------------------------------------------------------
>
>                 Key: HADOOP-9357
>                 URL: https://issues.apache.org/jira/browse/HADOOP-9357
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Andrew Wang
>            Assignee: Andrew Wang
>            Priority: Minor
>             Fix For: 2.0.4-alpha
>
>         Attachments: hadoop-9357-1.patch, hadoop-9357-2.patch, hadoop-9357-3.patch, hadoop-9357-testfixup.patch
>
>
> Currently, FileContext adheres rather strictly to RFC2396 when it comes to parsing absolute
URIs (URIs with a scheme). If a user asks for a URI like "hdfs:///tmp", FileContext will error
while FileSystem will add the authority of the default FS (e.g. turn it into "hdfs://defaultNN:port/tmp").

> This is technically correct, but FileSystem's behavior is nicer for users and okay based
on 5.2.3 in the RFC, so lets do it in FileContext too:
> {noformat}
> For backwards
> compatibility, an implementation may work around such references
> by removing the scheme if it matches that of the base URI and the
> scheme is known to always use the  syntax.  The parser
> can then continue with the steps below for the remainder of the
> reference components.  Validating parsers should mark such a
> misformed relative reference as an error.
> {noformat}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message