hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Andrew Wang (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-9357) Fallback to default authority if not specified in FileContext
Date Tue, 02 Apr 2013 03:41:16 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-9357?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13619485#comment-13619485

Andrew Wang commented on HADOOP-9357:

Hi Sanjay,

Thanks for taking a look, I understand and agree with most of your points. I created this
JIRA after some end user confusion when porting an app from FileSystem to FileContext. I felt
that URIs like {{hdfs:///tmp}} should be discouraged but supported, since they are supported
by FileSystem, and this case is specifically mentioned in the RFC as an okay fallback.

A couple potentially clarifying points:

bq. A user wanting /tmp on the default file system should say "/tmp" and not "hdfs:///tmp".

This code triggers when the schema matches the schema of the default FS. {{hdfs:///tmp}} will
only work if the default FS is an HDFS instance.

bq. While the notion of a default file system is very appropriate, the notion of default authority
per scheme does not make sense, especially if it is derived from the default file system.

There is not a default authority per scheme, as per the previous.

bq. In a multi-cluster environment, getting in the default authority/port from the default
file system can lead to accessing the files at the wrong HDFS cluster/volume...
bq. If you encourage the use of hdfs:///tmp then as you change your default file system from
hdfs to localfs...

I agree there could be ambiguity if you have multiple FS with the same schema, or change the
default FS.

Moving toward a meta question, we already have all these problems with FileSystem, which is
the dominant interface to HDFS. Have we encountered these problems in FileSystem with ambiguity
and changing the default FS? I at least haven't heard of any, but I did see the mail that
led to this JIRA.
> Fallback to default authority if not specified in FileContext
> -------------------------------------------------------------
>                 Key: HADOOP-9357
>                 URL: https://issues.apache.org/jira/browse/HADOOP-9357
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Andrew Wang
>            Assignee: Andrew Wang
>            Priority: Minor
>             Fix For: 2.0.4-alpha
>         Attachments: hadoop-9357-1.patch, hadoop-9357-2.patch, hadoop-9357-3.patch, hadoop-9357-testfixup.patch
> Currently, FileContext adheres rather strictly to RFC2396 when it comes to parsing absolute
URIs (URIs with a scheme). If a user asks for a URI like "hdfs:///tmp", FileContext will error
while FileSystem will add the authority of the default FS (e.g. turn it into "hdfs://defaultNN:port/tmp").

> This is technically correct, but FileSystem's behavior is nicer for users and okay based
on 5.2.3 in the RFC, so lets do it in FileContext too:
> {noformat}
> For backwards
> compatibility, an implementation may work around such references
> by removing the scheme if it matches that of the base URI and the
> scheme is known to always use the  syntax.  The parser
> can then continue with the steps below for the remainder of the
> reference components.  Validating parsers should mark such a
> misformed relative reference as an error.
> {noformat}

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

View raw message