hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Aaron Tokhy (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-9565) Add a Blobstore interface to add to blobstore FileSystems
Date Wed, 16 Mar 2016 22:34:34 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-9565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15198324#comment-15198324

Aaron Tokhy commented on HADOOP-9565:

FileOutputCommitter recently introduced a notion of 'algorithm version' (through mapreduce.fileoutputcommitter.algorithm.version).
 Perhaps the change to FileOutputCommitter would involve adding another algorithm version
(3) for performing a direct commit, separate from this change?  FileOutputCommitter could
ideally be split into 3 subclasses today, 1 per algorithm version.

Another boolean configuration could be added to mapred-default.xml, called mapreduce.fileoutputcommitter.infer.algorithm.version,
which would select the appropriate FileOutputCommitter algorithm based on the FileSystem for
the Path in mapreduce.output.fileoutputformat.outputdir.  MRAppMaster could then select the
appropriate algorithm depending on the semantics available in the target filesystem.

It might also make some sense to break up FileOutputCommitter algorithms into separate classes
to accommodate for these changes, since the algorithm used may not work well on some FileSystem
implementations (especially due to the attempts of performing atomic renames, on FileSystem
implementations where directory renames are not atomic).

> Add a Blobstore interface to add to blobstore FileSystems
> ---------------------------------------------------------
>                 Key: HADOOP-9565
>                 URL: https://issues.apache.org/jira/browse/HADOOP-9565
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs, fs/s3, fs/swift
>    Affects Versions: 2.6.0
>            Reporter: Steve Loughran
>            Assignee: Thomas Demoor
>              Labels: BB2015-05-TBR
>         Attachments: HADOOP-9565-001.patch, HADOOP-9565-002.patch, HADOOP-9565-003.patch,
HADOOP-9565-004.patch, HADOOP-9565-005.patch
> We can make the fact that some {{FileSystem}} implementations are really blobstores,
with different atomicity and consistency guarantees, by adding a {{Blobstore}} interface to
add to them. 
> This could also be a place to add a {{Copy(Path,Path)}} method, assuming that all blobstores
implement at server-side copy operation as a substitute for rename.

This message was sent by Atlassian JIRA

View raw message