hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Andrew Wang (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-10919) Copy command should preserve raw.* namespace extended attributes
Date Tue, 12 Aug 2014 18:55:13 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-10919?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14094500#comment-14094500

Andrew Wang commented on HADOOP-10919:

Hi Sanjay,

Could we define the requirements for "transparent"? Right now it's transparent in that distcp
will decrypt when it reads from the normal path. This is what all existing distcp scripts
will be doing, copying to and from normal paths. It's less efficient since it involves decryption,
and results in different bytes-on-disk on the destination (either because it's unencrypted,
or it's given a different EDEK), but it's a reasonable and sometimes desirable behavior. Using
the /.reserved/raw paths is a way of doing a direct byte-to-byte identical copy, which is
also a sometimes desirable behavior.

It sounds like you want the direct byte-to-byte copy to be the default, but remember that
it's an API with sharp edges, many of which are laid out in the doc. /.r/r is also superuser
only, since it lets you muck directly with the raw xattrs. This means we can't transparently
add the /.r/r prefix if the distcp runs as a normal user. Because of all this, we decided
to implement the current, safer behavior.

Does this sound reasonable?

> Copy command should preserve raw.* namespace extended attributes
> ----------------------------------------------------------------
>                 Key: HADOOP-10919
>                 URL: https://issues.apache.org/jira/browse/HADOOP-10919
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 3.0.0
>            Reporter: Charles Lamb
>            Assignee: Charles Lamb
>             Fix For: fs-encryption (HADOOP-10150 and HDFS-6134)
>         Attachments: HADOOP-10919.001.patch, HADOOP-10919.002.patch
> Refer to the doc attached to HDFS-6509 for background.
> Like distcp -p (see MAPREDUCE-6007), the copy command also needs to preserve extended
attributes in the raw.* namespace by default whenever the src and target are in /.reserved/raw.
To not preserve raw xattrs, don't specify /.reserved/raw in either the src or target. 

This message was sent by Atlassian JIRA

View raw message