falcon-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Balu Vellanki (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (FALCON-1343) validation of read/write endpoints is not reliable
Date Tue, 22 Sep 2015 15:59:04 GMT

     [ https://issues.apache.org/jira/browse/FALCON-1343?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Balu Vellanki updated FALCON-1343:
----------------------------------
    Description: 
A read/write endpoint is currently validated by creating a filesystem with the endpoint url.

{code}
            HadoopClientFactory.get().createProxiedFileSystem(conf);
{code}

This is not sufficient validation for read/write endpoint. It wont catch any typos in the
url used for the endpoint. The  validation of cluster will then fail in validateLocations(...)
method, and will throw an exception that can confuse the user.  Better validation would be
to check if "/" exists in HDFS. 

  was:
A read/write endpoint is currently validated by creating a filesystem with the endpoint url.

{code}
            HadoopClientFactory.get().createProxiedFileSystem(conf);
{code}

I confirmed with a HDFS team member that it is not sufficient validation.  Ideally check if
the end user has access by doing atleast a list /tmp call after creating proxiedFileSystem.



> validation of read/write endpoints is not reliable
> --------------------------------------------------
>
>                 Key: FALCON-1343
>                 URL: https://issues.apache.org/jira/browse/FALCON-1343
>             Project: Falcon
>          Issue Type: Sub-task
>          Components: general
>            Reporter: Balu Vellanki
>            Assignee: Balu Vellanki
>             Fix For: 0.8
>
>         Attachments: FALCON-1343-v1.patch, FALCON-1343.patch
>
>
> A read/write endpoint is currently validated by creating a filesystem with the endpoint
url. 
> {code}
>             HadoopClientFactory.get().createProxiedFileSystem(conf);
> {code}
> This is not sufficient validation for read/write endpoint. It wont catch any typos in
the url used for the endpoint. The  validation of cluster will then fail in validateLocations(...)
method, and will throw an exception that can confuse the user.  Better validation would be
to check if "/" exists in HDFS. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message