hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Aaron Fabbri (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-13448) S3Guard: Define MetadataStore interface.
Date Wed, 10 Aug 2016 17:44:20 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-13448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15415658#comment-15415658
] 

Aaron Fabbri commented on HADOOP-13448:
---------------------------------------

To start the interface discussion:

I was looking at the two ConsistentStore/MetadataStore interfaces we posted to HADOOP-13345,
and had a question:  Why does your {{DynamoDBConsistentStore#save()}} implementation walk
the path to the root and save all ancestor paths as well?  I understand that this is mirroring
what the S3A client does.  I guess I preferred leaving the enumeration of ancestor directories
that need to be created to the client.  The case I'm thinking of is a workload that does this:

{code}
path = "/dirs"
for x in range(0,10) :
  path += "/dir" + str(x)
  s3aFS.mkdirs(path)
{code}

In this case, we're making N directories that get deeper and deeper with the same ancestors.
 This seems like an theta(N^2) cost with your prototype, but only N cost with ours.  The downside
to our approach is that the client code has to explicitly track the missing ancestors that
need to be passed to the MetadataStore.  In our patch it looked like this at the end of {{innerMkdirs()}}:

{code}
        } catch (FileNotFoundException fnfe) {
          instrumentation.errorIgnored();
          if (dirsToCreate != null)
            dirsToCreate.add(fPart);
        }
        fPart = fPart.getParent();
      } while (fPart != null);

      String key = pathToKey(f);
      createFakeDirectory(key);
      metadataStoreMkdirs(dirsToCreate, permission);
      return true;
    }
  }
{code}

I might be missing something here, let me know what you think.

> S3Guard: Define MetadataStore interface.
> ----------------------------------------
>
>                 Key: HADOOP-13448
>                 URL: https://issues.apache.org/jira/browse/HADOOP-13448
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: fs/s3
>            Reporter: Chris Nauroth
>            Assignee: Chris Nauroth
>
> Define the common interface for metadata store operations.  This is the interface that
any metadata back-end must implement in order to integrate with S3Guard.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


Mime
View raw message