hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Chris Nauroth (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-12635) Adding Append API support for WASB
Date Mon, 11 Jan 2016 23:52:39 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-12635?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15092955#comment-15092955

Chris Nauroth commented on HADOOP-12635:

[~dchickabasapa], I thought of another potential problem with the current patch.

The code I've seen so far is focused on the {{append}} API.  I haven't seen any code changes
in the {{create}} code path, so WASB {{create}} calls would not coordinate on the lease with
the {{append}} calls.  This means that it would be possible for process A to open a file for
{{create}}, and concurrently process B opens the same file for {{append}}.  Both processes
would write data to their respective streams, and I guess ultimately both could flush and
commit blocks to Azure Storage.  This differs from HDFS, which issues a lease on a path for
both {{create}} and {{append}} calls, thus ensuring single-writer semantics across both {{create}}
and {{append}}.  In this same example executing against HDFS, one of the processes would have
succeeded, and the other would have received an error.

If your goal is to match HDFS semantics fully, then the patch would need to do something about
the {{create}} calls too.

> Adding Append API support for WASB
> ----------------------------------
>                 Key: HADOOP-12635
>                 URL: https://issues.apache.org/jira/browse/HADOOP-12635
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: azure
>    Affects Versions: 2.7.1
>            Reporter: Dushyanth
>            Assignee: Dushyanth
>         Attachments: Append API.docx, HADOOP-12635.001.patch, HADOOP-12635.002.patch
> Currently the WASB implementation of the HDFS interface does not support Append API.
This JIRA is added to design and implement the Append API support to WASB. The intended support
for Append would only support a single writer.  

This message was sent by Atlassian JIRA

View raw message