hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Aaron Fabbri (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (HADOOP-11572) s3a delete() operation fails during a concurrent delete of child entries
Date Wed, 17 May 2017 21:03:04 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-11572?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16014765#comment-16014765
] 

Aaron Fabbri edited comment on HADOOP-11572 at 5/17/17 9:02 PM:
----------------------------------------------------------------

edit: removing comment.. I reviewed the old v1 patch. 


was (Author: fabbri):
{code}
@@ -688,12 +693,32 @@ public boolean rename(Path src, Path dst) throws IOException {
    */
   private void removeKeys(List<DeleteObjectsRequest.KeyVersion> keysToDelete,
           boolean clearKeys) {
+    List<DeleteObjectsRequest.KeyVersion> keysToDeleteSingly;
     if (enableMultiObjectsDelete) {
       DeleteObjectsRequest deleteRequest
           = new DeleteObjectsRequest(bucket).withKeys(keysToDelete);
       s3.deleteObjects(deleteRequest);
       statistics.incrementWriteOps(1);
+      try {
+        s3.deleteObjects(deleteRequest);
+        keysToDeleteSingly = null;
{code}

It looks like you need to remove the call to {{s3.deleteObjects(..)}} before your try block,
and maybe move the write ops increment down?

{code}
+      } catch (MultiObjectDeleteException e) {
+        // partial delete of files
+        List<MultiObjectDeleteException.DeleteError> errors = e.getErrors();
+        LOG.warn("Partial failure of delete, {} objects remaining",
+            errors.size());
+        keysToDeleteSingly = new ArrayList<>(errors.size());
+        for (MultiObjectDeleteException.DeleteError error : errors) {
+          keysToDeleteSingly.add(
+              new DeleteObjectsRequest.KeyVersion(error.getKey(),
+                  error.getVersionId()));
+        }
+      }
     } else {
+      keysToDeleteSingly = keysToDelete;
+    }
+
+    if (keysToDeleteSingly != null && !keysToDeleteSingly.isEmpty()) {
       int writeops = 0;
  
       for (DeleteObjectsRequest.KeyVersion keyVersion : keysToDelete) {
{code}
Should this loop be using {{keysToDeleteSingly}} instead of {{keysToDelete}}?

> s3a delete() operation fails during a concurrent delete of child entries
> ------------------------------------------------------------------------
>
>                 Key: HADOOP-11572
>                 URL: https://issues.apache.org/jira/browse/HADOOP-11572
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: fs/s3
>    Affects Versions: 2.6.0
>            Reporter: Steve Loughran
>            Assignee: Steve Loughran
>         Attachments: HADOOP-11572-001.patch, HADOOP-11572-branch-2-002.patch, HADOOP-11572-branch-2-003.patch
>
>
> Reviewing the code, s3a has the problem raised in HADOOP-6688: deletion of a child entry
during a recursive directory delete is propagated as an exception, rather than ignored as
a detail which idempotent operations should just ignore.
> the exception should be caught and, if a file not found problem, logged rather than propagated



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


Mime
View raw message