hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sanjay Radia (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-10741) A lightweight WebHDFS client library
Date Thu, 02 Oct 2014 01:33:34 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-10741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14155967#comment-14155967

Sanjay Radia commented on HADOOP-10741:

 I see part of the counter argument being that folks using rest are doing it for one of two
1) Protocol compatibility - this was the orignal motivation in the past when HDFS protocols
were not compatible across some versions, This has been fixed.
2) Want a lightweight client that is independent of any version of HDFS. However as Mohammad
has pointed out in his description, ustomer using web hdfs rest protocol find that managing
failure, auth, etc is painful, Hence a library would help.
I can see Andrew's argument of putting it outside Hadoop common to better satisfy (2). We
can decide the exact mechanism to distribute this library later.  
Note the goal of this library is *not* another FS API but a client side library that wraps
hdfs's rest protocol. It is valid question to see if this API should mimic that actual Hadoop
Mohammad please post the patch. We will figure out the mechanism of distributing that library
separately. Thanks.

> A lightweight WebHDFS client library
> ------------------------------------
>                 Key: HADOOP-10741
>                 URL: https://issues.apache.org/jira/browse/HADOOP-10741
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: tools
>            Reporter: Tsz Wo Nicholas Sze
>            Assignee: Mohammad Kamrul Islam
> One of the motivations for creating WebHDFS is for applications connecting to HDFS from
outside the cluster.  In order to do so, users have to either
> # install Hadoop and use WebHdfsFileSsytem, or
> # develop their own client using the WebHDFS REST API.
> For #1, it is very difficult to manage and unnecessarily complicated for other applications
since Hadoop is not a lightweight library.  For #2, it is not easy to deal with security and
handle transient errors.
> Therefore, we propose adding a lightweight WebHDFS client as a separated library which
does not depend on Common and HDFS.  The client can be packaged as a standalone jar.  Other
applications simply add the jar to their classpath for using it.

This message was sent by Atlassian JIRA

View raw message