jclouds-notifications mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "roded (Jira)" <j...@apache.org>
Subject [jira] [Updated] (JCLOUDS-1549) Multipart upload fails on AWS S3 buckets with Object Lock and retention
Date Tue, 21 Jul 2020 16:20:00 GMT

     [ https://issues.apache.org/jira/browse/JCLOUDS-1549?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

roded updated JCLOUDS-1549:
---------------------------
    Description: 
AWS S3 buckets with object lock and a default object retention period (governance/compliance)
require the content MD5 header for PUT operations ([https://docs.aws.amazon.com/AmazonS3/latest/dev/object-lock-overview.html])
 When multipart uploading a blob into such a bucket, a response is returned indicating that
the content MD5 header is missing.

The following snippet:
{code:java}
BlobStoreContext blobStoreContext = ContextBuilder.newBuilder("aws-s3")
    .endpoint("https://s3.amazonaws.com")
    .credentials("...", "...")
    .buildView(BlobStoreContext.class);
BlobStore blobStore = blobStoreContext.getBlobStore();
ByteSource payloadByteSource = ByteSource.wrap("payloaddata".getBytes(StandardCharsets.UTF_8));
Blob blob = blobStore.blobBuilder("multipart-uploaded-blob")
    .userMetadata(Collections.emptyMap())
    .payload(payloadByteSource)
    .contentLength(payloadByteSource.size())
    .contentMD5(payloadByteSource.hash(Hashing.md5()))
    .build();
blobStore.putBlob("my-temp-object-locked-bucket", blob, PutOptions.Builder.multipart());
blobStoreContext.close(); 
{code}
 

Results in the exception:
{code:java}
com.google.common.util.concurrent.UncheckedExecutionException: org.jclouds.aws.AWSResponseException:
request PUT https://my-temp-object-locked-bucket.s3-eu-central-1.amazonaws.com/multipart-uploaded-blob?partNumber=1&uploadId=7SXM.6rCAh_zyPYoF1KPyc86niKrkvUqoke00mEbJrAj9pdmp8Cf0cK6uGbxVBjEJNXkO8JVC6Yb5aQKI30WP4eohjdfFQOtY888JlUm6eOFKWy8lsmNZ5fBgRcPCdMD
HTTP/1.1 failed with code 400, error: AWSError{requestId='91FBACF8DF0AE5EB', requestToken='1ce75Bg73qNTIjUfGiI4QjEIarPEEKm37NMNvt7dLAM68DHYsqPahR1JKVOobkPHX/fSidHbdmM=',
code='InvalidRequest', message='Content-MD5 HTTP header is required for Put Part requests
with Object Lock parameters', context='{HostId=1ce75Bg73qNTIjUfGiI4QjEIarPEEKm37NMNvt7dLAM68DHYsqPahR1JKVOobkPHX/fSidHbdmM=}'}
at com.google.common.util.concurrent.Futures.wrapAndThrowUnchecked(Futures.java:1525)
	at com.google.common.util.concurrent.Futures.getUnchecked(Futures.java:1511)
	at org.jclouds.blobstore.internal.BaseBlobStore.putMultipartBlob(BaseBlobStore.java:395)
	at org.jclouds.blobstore.internal.BaseBlobStore.putMultipartBlob(BaseBlobStore.java:349)
	at org.jclouds.aws.s3.blobstore.AWSS3BlobStore.putBlob(AWSS3BlobStore.java:79)
	at io.model9.backup.common.objectstorage.MultipartMd5Test.multipartMd5(MultipartMd5Test.java:34)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
	at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
	at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
	at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
	at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
	at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
	at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
	at com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:69)
	at com.intellij.rt.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:33)
	at com.intellij.rt.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:220)
	at com.intellij.rt.junit.JUnitStarter.main(JUnitStarter.java:53)
Caused by: org.jclouds.aws.AWSResponseException: request PUT https://my-temp-object-locked-bucket.s3-eu-central-1.amazonaws.com/multipart-uploaded-blob?partNumber=1&uploadId=7SXM.6rCAh_zyPYoF1KPyc86niKrkvUqoke00mEbJrAj9pdmp8Cf0cK6uGbxVBjEJNXkO8JVC6Yb5aQKI30WP4eohjdfFQOtY888JlUm6eOFKWy8lsmNZ5fBgRcPCdMD
HTTP/1.1 failed with code 400, error: AWSError{requestId='91FBACF8DF0AE5EB', requestToken='1ce75Bg73qNTIjUfGiI4QjEIarPEEKm37NMNvt7dLAM68DHYsqPahR1JKVOobkPHX/fSidHbdmM=',
code='InvalidRequest', message='Content-MD5 HTTP header is required for Put Part requests
with Object Lock parameters', context='{HostId=1ce75Bg73qNTIjUfGiI4QjEIarPEEKm37NMNvt7dLAM68DHYsqPahR1JKVOobkPHX/fSidHbdmM=}'}
	at org.jclouds.aws.handlers.ParseAWSErrorFromXmlContent.handleError(ParseAWSErrorFromXmlContent.java:76)
	at org.jclouds.http.handlers.DelegatingErrorHandler.handleError(DelegatingErrorHandler.java:65)
	at org.jclouds.http.internal.BaseHttpCommandExecutorService.shouldContinue(BaseHttpCommandExecutorService.java:138)
	at org.jclouds.http.internal.BaseHttpCommandExecutorService.invoke(BaseHttpCommandExecutorService.java:107)
	at org.jclouds.rest.internal.InvokeHttpMethod.invoke(InvokeHttpMethod.java:91)
	at org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:74)
	at org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:45)
	at org.jclouds.rest.internal.DelegatesToInvocationFunction.handle(DelegatesToInvocationFunction.java:156)
	at org.jclouds.rest.internal.DelegatesToInvocationFunction.invoke(DelegatesToInvocationFunction.java:123)
	at com.sun.proxy.$Proxy44.uploadPart(Unknown Source)
	at org.jclouds.s3.blobstore.S3BlobStore.uploadMultipartPart(S3BlobStore.java:391)
	at org.jclouds.blobstore.internal.BaseBlobStore$BlobUploader.call(BaseBlobStore.java:415)
	at org.jclouds.blobstore.internal.BaseBlobStore$BlobUploader.call(BaseBlobStore.java:402)
	at java.util.concurrent.FutureTask.run$$$capture(FutureTask.java:266)
	at java.util.concurrent.FutureTask.run(FutureTask.java)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
	at java.lang.Thread.getStackTrace(Thread.java:1559)
	at org.jclouds.blobstore.internal.BaseBlobStore.putMultipartBlob(BaseBlobStore.java:393)
	... 25 more
{code}
 

Here are the requests (URL and headers) initiated by JClouds in the above snippet:
{code:java}
GET >> https://s3.amazonaws.com/my-temp-object-locked-bucket?location
{Host=[s3.amazonaws.com], x-amz-content-sha256=[e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855],
X-Amz-Date=[20200721T160746Z], Authorization=[AWS4-HMAC-SHA256 Credential=..., SignedHeaders=host;x-amz-content-sha256;x-amz-date,
Signature=db96f64af0bc4457ceda679805b6bb70656494b0ae5ae4d6501f1b19847dc31a]}
POST >> https://my-temp-object-locked-bucket.s3-eu-central-1.amazonaws.com/multipart-uploaded-blob?uploads
{Content-Type=[application/unknown], Content-MD5=[nirLkofY1bN5hCACAirgRA==], Host=[my-temp-object-locked-bucket.s3-eu-central-1.amazonaws.com],
x-amz-content-sha256=[e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855], X-Amz-Date=[20200721T160746Z],
Authorization=[AWS4-HMAC-SHA256 Credential=..., SignedHeaders=content-md5;content-type;host;x-amz-content-sha256;x-amz-date,
Signature=fdb6a51c9cfbe116f8da033e079d4a9033a63b1d7a3abe8256546b2363f2dddb]}
PUT >> https://my-temp-object-locked-bucket.s3-eu-central-1.amazonaws.com/multipart-uploaded-blob?partNumber=1&uploadId=7SXM.6rCAh_zyPYoF1KPyc86niKrkvUqoke00mEbJrAj9pdmp8Cf0cK6uGbxVBjEJNXkO8JVC6Yb5aQKI30WP4eohjdfFQOtY888JlUm6eOFKWy8lsmNZ5fBgRcPCdMD
{Content-Type=[application/unknown], Content-Length=[11], Host=[my-temp-object-locked-bucket.s3-eu-central-1.amazonaws.com],
x-amz-content-sha256=[9e8f1b13c357dd00b99dd1760be67a3fdeb3ed1e3605307e549ab74c483a43fa], X-Amz-Date=[20200721T160746Z],
Authorization=[AWS4-HMAC-SHA256 Credential=.., SignedHeaders=content-length;content-type;host;x-amz-content-sha256;x-amz-date,
Signature=9aec5a51bbffaa95321a054d8d6aad40b7c7d87d6967562592f8aaf51bb5e7aa]}
DELETE >> https://my-temp-object-locked-bucket.s3-eu-central-1.amazonaws.com/multipart-uploaded-blob?uploadId=7SXM.6rCAh_zyPYoF1KPyc86niKrkvUqoke00mEbJrAj9pdmp8Cf0cK6uGbxVBjEJNXkO8JVC6Yb5aQKI30WP4eohjdfFQOtY888JlUm6eOFKWy8lsmNZ5fBgRcPCdMD
{Host=[my-temp-object-locked-bucket.s3-eu-central-1.amazonaws.com], x-amz-content-sha256=[e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855],
X-Amz-Date=[20200721T160746Z], Authorization=[AWS4-HMAC-SHA256 Credential=..., SignedHeaders=host;x-amz-content-sha256;x-amz-date,
Signature=f2e53347d054d7c4a08eb1d62b1ff828d90135d0563d85ee8547626788dfd290]}
{code}
It looks like the blob's MD5 hash is added to the POST to initiate the multipart upload but
is missing from the part's PUT itself.

 

 

  was:
AWS S3 buckets with object lock and a default object retention period (governance/compliance)
require the content MD5 header for PUT operations ([https://docs.aws.amazon.com/AmazonS3/latest/dev/object-lock-overview.html])
 When multipart uploading a blob into such a bucket, a response is returned indicating that
the content MD5 header is missing.

The following snippet:
{code:java}
BlobStoreContext blobStoreContext = ContextBuilder.newBuilder("aws-s3")
    .endpoint("https://s3.amazonaws.com")
    .credentials("...", "...")
    .buildView(BlobStoreContext.class);
BlobStore blobStore = blobStoreContext.getBlobStore();
ByteSource payloadByteSource = ByteSource.wrap("payloaddata".getBytes(StandardCharsets.UTF_8));
Blob blob = blobStore.blobBuilder("multipart-uploaded-blob")
    .userMetadata(Collections.emptyMap())
    .payload(payloadByteSource)
    .contentLength(payloadByteSource.size())
    .contentMD5(payloadByteSource.hash(Hashing.md5()))
    .build();
blobStore.putBlob("my-temp-object-locked-bucket", blob, PutOptions.Builder.multipart());
blobStoreContext.close(); 
{code}
Results in the exception:

 
{code:java}
com.google.common.util.concurrent.UncheckedExecutionException: org.jclouds.aws.AWSResponseException:
request PUT https://my-temp-object-locked-bucket.s3-eu-central-1.amazonaws.com/multipart-uploaded-blob?partNumber=1&uploadId=7SXM.6rCAh_zyPYoF1KPyc86niKrkvUqoke00mEbJrAj9pdmp8Cf0cK6uGbxVBjEJNXkO8JVC6Yb5aQKI30WP4eohjdfFQOtY888JlUm6eOFKWy8lsmNZ5fBgRcPCdMD
HTTP/1.1 failed with code 400, error: AWSError{requestId='91FBACF8DF0AE5EB', requestToken='1ce75Bg73qNTIjUfGiI4QjEIarPEEKm37NMNvt7dLAM68DHYsqPahR1JKVOobkPHX/fSidHbdmM=',
code='InvalidRequest', message='Content-MD5 HTTP header is required for Put Part requests
with Object Lock parameters', context='{HostId=1ce75Bg73qNTIjUfGiI4QjEIarPEEKm37NMNvt7dLAM68DHYsqPahR1JKVOobkPHX/fSidHbdmM=}'}
at com.google.common.util.concurrent.Futures.wrapAndThrowUnchecked(Futures.java:1525)
	at com.google.common.util.concurrent.Futures.getUnchecked(Futures.java:1511)
	at org.jclouds.blobstore.internal.BaseBlobStore.putMultipartBlob(BaseBlobStore.java:395)
	at org.jclouds.blobstore.internal.BaseBlobStore.putMultipartBlob(BaseBlobStore.java:349)
	at org.jclouds.aws.s3.blobstore.AWSS3BlobStore.putBlob(AWSS3BlobStore.java:79)
	at io.model9.backup.common.objectstorage.MultipartMd5Test.multipartMd5(MultipartMd5Test.java:34)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
	at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
	at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
	at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
	at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
	at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
	at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
	at com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:69)
	at com.intellij.rt.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:33)
	at com.intellij.rt.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:220)
	at com.intellij.rt.junit.JUnitStarter.main(JUnitStarter.java:53)
Caused by: org.jclouds.aws.AWSResponseException: request PUT https://my-temp-object-locked-bucket.s3-eu-central-1.amazonaws.com/multipart-uploaded-blob?partNumber=1&uploadId=7SXM.6rCAh_zyPYoF1KPyc86niKrkvUqoke00mEbJrAj9pdmp8Cf0cK6uGbxVBjEJNXkO8JVC6Yb5aQKI30WP4eohjdfFQOtY888JlUm6eOFKWy8lsmNZ5fBgRcPCdMD
HTTP/1.1 failed with code 400, error: AWSError{requestId='91FBACF8DF0AE5EB', requestToken='1ce75Bg73qNTIjUfGiI4QjEIarPEEKm37NMNvt7dLAM68DHYsqPahR1JKVOobkPHX/fSidHbdmM=',
code='InvalidRequest', message='Content-MD5 HTTP header is required for Put Part requests
with Object Lock parameters', context='{HostId=1ce75Bg73qNTIjUfGiI4QjEIarPEEKm37NMNvt7dLAM68DHYsqPahR1JKVOobkPHX/fSidHbdmM=}'}
	at org.jclouds.aws.handlers.ParseAWSErrorFromXmlContent.handleError(ParseAWSErrorFromXmlContent.java:76)
	at org.jclouds.http.handlers.DelegatingErrorHandler.handleError(DelegatingErrorHandler.java:65)
	at org.jclouds.http.internal.BaseHttpCommandExecutorService.shouldContinue(BaseHttpCommandExecutorService.java:138)
	at org.jclouds.http.internal.BaseHttpCommandExecutorService.invoke(BaseHttpCommandExecutorService.java:107)
	at org.jclouds.rest.internal.InvokeHttpMethod.invoke(InvokeHttpMethod.java:91)
	at org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:74)
	at org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:45)
	at org.jclouds.rest.internal.DelegatesToInvocationFunction.handle(DelegatesToInvocationFunction.java:156)
	at org.jclouds.rest.internal.DelegatesToInvocationFunction.invoke(DelegatesToInvocationFunction.java:123)
	at com.sun.proxy.$Proxy44.uploadPart(Unknown Source)
	at org.jclouds.s3.blobstore.S3BlobStore.uploadMultipartPart(S3BlobStore.java:391)
	at org.jclouds.blobstore.internal.BaseBlobStore$BlobUploader.call(BaseBlobStore.java:415)
	at org.jclouds.blobstore.internal.BaseBlobStore$BlobUploader.call(BaseBlobStore.java:402)
	at java.util.concurrent.FutureTask.run$$$capture(FutureTask.java:266)
	at java.util.concurrent.FutureTask.run(FutureTask.java)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
	at java.lang.Thread.getStackTrace(Thread.java:1559)
	at org.jclouds.blobstore.internal.BaseBlobStore.putMultipartBlob(BaseBlobStore.java:393)
	... 25 more
{code}
 

 

Here are the requests (URL and headers) initiated by JClouds in the above snippet:

 
{code:java}
GET >> https://s3.amazonaws.com/my-temp-object-locked-bucket?location
{Host=[s3.amazonaws.com], x-amz-content-sha256=[e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855],
X-Amz-Date=[20200721T160746Z], Authorization=[AWS4-HMAC-SHA256 Credential=..., SignedHeaders=host;x-amz-content-sha256;x-amz-date,
Signature=db96f64af0bc4457ceda679805b6bb70656494b0ae5ae4d6501f1b19847dc31a]}
POST >> https://my-temp-object-locked-bucket.s3-eu-central-1.amazonaws.com/multipart-uploaded-blob?uploads
{Content-Type=[application/unknown], Content-MD5=[nirLkofY1bN5hCACAirgRA==], Host=[my-temp-object-locked-bucket.s3-eu-central-1.amazonaws.com],
x-amz-content-sha256=[e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855], X-Amz-Date=[20200721T160746Z],
Authorization=[AWS4-HMAC-SHA256 Credential=..., SignedHeaders=content-md5;content-type;host;x-amz-content-sha256;x-amz-date,
Signature=fdb6a51c9cfbe116f8da033e079d4a9033a63b1d7a3abe8256546b2363f2dddb]}
PUT >> https://my-temp-object-locked-bucket.s3-eu-central-1.amazonaws.com/multipart-uploaded-blob?partNumber=1&uploadId=7SXM.6rCAh_zyPYoF1KPyc86niKrkvUqoke00mEbJrAj9pdmp8Cf0cK6uGbxVBjEJNXkO8JVC6Yb5aQKI30WP4eohjdfFQOtY888JlUm6eOFKWy8lsmNZ5fBgRcPCdMD
{Content-Type=[application/unknown], Content-Length=[11], Host=[my-temp-object-locked-bucket.s3-eu-central-1.amazonaws.com],
x-amz-content-sha256=[9e8f1b13c357dd00b99dd1760be67a3fdeb3ed1e3605307e549ab74c483a43fa], X-Amz-Date=[20200721T160746Z],
Authorization=[AWS4-HMAC-SHA256 Credential=.., SignedHeaders=content-length;content-type;host;x-amz-content-sha256;x-amz-date,
Signature=9aec5a51bbffaa95321a054d8d6aad40b7c7d87d6967562592f8aaf51bb5e7aa]}
DELETE >> https://my-temp-object-locked-bucket.s3-eu-central-1.amazonaws.com/multipart-uploaded-blob?uploadId=7SXM.6rCAh_zyPYoF1KPyc86niKrkvUqoke00mEbJrAj9pdmp8Cf0cK6uGbxVBjEJNXkO8JVC6Yb5aQKI30WP4eohjdfFQOtY888JlUm6eOFKWy8lsmNZ5fBgRcPCdMD
{Host=[my-temp-object-locked-bucket.s3-eu-central-1.amazonaws.com], x-amz-content-sha256=[e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855],
X-Amz-Date=[20200721T160746Z], Authorization=[AWS4-HMAC-SHA256 Credential=..., SignedHeaders=host;x-amz-content-sha256;x-amz-date,
Signature=f2e53347d054d7c4a08eb1d62b1ff828d90135d0563d85ee8547626788dfd290]}
{code}
It looks like the blob's MD5 hash is added to the POST to initiate the multipart upload but
is missing from the part's PUT itself.

 

 


> Multipart upload fails on AWS S3 buckets with Object Lock and retention
> -----------------------------------------------------------------------
>
>                 Key: JCLOUDS-1549
>                 URL: https://issues.apache.org/jira/browse/JCLOUDS-1549
>             Project: jclouds
>          Issue Type: Bug
>          Components: jclouds-blobstore
>    Affects Versions: 2.2.0
>            Reporter: roded
>            Priority: Major
>
> AWS S3 buckets with object lock and a default object retention period (governance/compliance)
require the content MD5 header for PUT operations ([https://docs.aws.amazon.com/AmazonS3/latest/dev/object-lock-overview.html])
>  When multipart uploading a blob into such a bucket, a response is returned indicating
that the content MD5 header is missing.
> The following snippet:
> {code:java}
> BlobStoreContext blobStoreContext = ContextBuilder.newBuilder("aws-s3")
>     .endpoint("https://s3.amazonaws.com")
>     .credentials("...", "...")
>     .buildView(BlobStoreContext.class);
> BlobStore blobStore = blobStoreContext.getBlobStore();
> ByteSource payloadByteSource = ByteSource.wrap("payloaddata".getBytes(StandardCharsets.UTF_8));
> Blob blob = blobStore.blobBuilder("multipart-uploaded-blob")
>     .userMetadata(Collections.emptyMap())
>     .payload(payloadByteSource)
>     .contentLength(payloadByteSource.size())
>     .contentMD5(payloadByteSource.hash(Hashing.md5()))
>     .build();
> blobStore.putBlob("my-temp-object-locked-bucket", blob, PutOptions.Builder.multipart());
> blobStoreContext.close(); 
> {code}
>  
> Results in the exception:
> {code:java}
> com.google.common.util.concurrent.UncheckedExecutionException: org.jclouds.aws.AWSResponseException:
request PUT https://my-temp-object-locked-bucket.s3-eu-central-1.amazonaws.com/multipart-uploaded-blob?partNumber=1&uploadId=7SXM.6rCAh_zyPYoF1KPyc86niKrkvUqoke00mEbJrAj9pdmp8Cf0cK6uGbxVBjEJNXkO8JVC6Yb5aQKI30WP4eohjdfFQOtY888JlUm6eOFKWy8lsmNZ5fBgRcPCdMD
HTTP/1.1 failed with code 400, error: AWSError{requestId='91FBACF8DF0AE5EB', requestToken='1ce75Bg73qNTIjUfGiI4QjEIarPEEKm37NMNvt7dLAM68DHYsqPahR1JKVOobkPHX/fSidHbdmM=',
code='InvalidRequest', message='Content-MD5 HTTP header is required for Put Part requests
with Object Lock parameters', context='{HostId=1ce75Bg73qNTIjUfGiI4QjEIarPEEKm37NMNvt7dLAM68DHYsqPahR1JKVOobkPHX/fSidHbdmM=}'}
at com.google.common.util.concurrent.Futures.wrapAndThrowUnchecked(Futures.java:1525)
> 	at com.google.common.util.concurrent.Futures.getUnchecked(Futures.java:1511)
> 	at org.jclouds.blobstore.internal.BaseBlobStore.putMultipartBlob(BaseBlobStore.java:395)
> 	at org.jclouds.blobstore.internal.BaseBlobStore.putMultipartBlob(BaseBlobStore.java:349)
> 	at org.jclouds.aws.s3.blobstore.AWSS3BlobStore.putBlob(AWSS3BlobStore.java:79)
> 	at io.model9.backup.common.objectstorage.MultipartMd5Test.multipartMd5(MultipartMd5Test.java:34)
> 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 	at java.lang.reflect.Method.invoke(Method.java:498)
> 	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> 	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> 	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> 	at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> 	at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
> 	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> 	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> 	at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> 	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> 	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> 	at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> 	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> 	at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> 	at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
> 	at com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:69)
> 	at com.intellij.rt.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:33)
> 	at com.intellij.rt.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:220)
> 	at com.intellij.rt.junit.JUnitStarter.main(JUnitStarter.java:53)
> Caused by: org.jclouds.aws.AWSResponseException: request PUT https://my-temp-object-locked-bucket.s3-eu-central-1.amazonaws.com/multipart-uploaded-blob?partNumber=1&uploadId=7SXM.6rCAh_zyPYoF1KPyc86niKrkvUqoke00mEbJrAj9pdmp8Cf0cK6uGbxVBjEJNXkO8JVC6Yb5aQKI30WP4eohjdfFQOtY888JlUm6eOFKWy8lsmNZ5fBgRcPCdMD
HTTP/1.1 failed with code 400, error: AWSError{requestId='91FBACF8DF0AE5EB', requestToken='1ce75Bg73qNTIjUfGiI4QjEIarPEEKm37NMNvt7dLAM68DHYsqPahR1JKVOobkPHX/fSidHbdmM=',
code='InvalidRequest', message='Content-MD5 HTTP header is required for Put Part requests
with Object Lock parameters', context='{HostId=1ce75Bg73qNTIjUfGiI4QjEIarPEEKm37NMNvt7dLAM68DHYsqPahR1JKVOobkPHX/fSidHbdmM=}'}
> 	at org.jclouds.aws.handlers.ParseAWSErrorFromXmlContent.handleError(ParseAWSErrorFromXmlContent.java:76)
> 	at org.jclouds.http.handlers.DelegatingErrorHandler.handleError(DelegatingErrorHandler.java:65)
> 	at org.jclouds.http.internal.BaseHttpCommandExecutorService.shouldContinue(BaseHttpCommandExecutorService.java:138)
> 	at org.jclouds.http.internal.BaseHttpCommandExecutorService.invoke(BaseHttpCommandExecutorService.java:107)
> 	at org.jclouds.rest.internal.InvokeHttpMethod.invoke(InvokeHttpMethod.java:91)
> 	at org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:74)
> 	at org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:45)
> 	at org.jclouds.rest.internal.DelegatesToInvocationFunction.handle(DelegatesToInvocationFunction.java:156)
> 	at org.jclouds.rest.internal.DelegatesToInvocationFunction.invoke(DelegatesToInvocationFunction.java:123)
> 	at com.sun.proxy.$Proxy44.uploadPart(Unknown Source)
> 	at org.jclouds.s3.blobstore.S3BlobStore.uploadMultipartPart(S3BlobStore.java:391)
> 	at org.jclouds.blobstore.internal.BaseBlobStore$BlobUploader.call(BaseBlobStore.java:415)
> 	at org.jclouds.blobstore.internal.BaseBlobStore$BlobUploader.call(BaseBlobStore.java:402)
> 	at java.util.concurrent.FutureTask.run$$$capture(FutureTask.java:266)
> 	at java.util.concurrent.FutureTask.run(FutureTask.java)
> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> 	at java.lang.Thread.run(Thread.java:748)
> 	at java.lang.Thread.getStackTrace(Thread.java:1559)
> 	at org.jclouds.blobstore.internal.BaseBlobStore.putMultipartBlob(BaseBlobStore.java:393)
> 	... 25 more
> {code}
>  
> Here are the requests (URL and headers) initiated by JClouds in the above snippet:
> {code:java}
> GET >> https://s3.amazonaws.com/my-temp-object-locked-bucket?location
> {Host=[s3.amazonaws.com], x-amz-content-sha256=[e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855],
X-Amz-Date=[20200721T160746Z], Authorization=[AWS4-HMAC-SHA256 Credential=..., SignedHeaders=host;x-amz-content-sha256;x-amz-date,
Signature=db96f64af0bc4457ceda679805b6bb70656494b0ae5ae4d6501f1b19847dc31a]}
> POST >> https://my-temp-object-locked-bucket.s3-eu-central-1.amazonaws.com/multipart-uploaded-blob?uploads
> {Content-Type=[application/unknown], Content-MD5=[nirLkofY1bN5hCACAirgRA==], Host=[my-temp-object-locked-bucket.s3-eu-central-1.amazonaws.com],
x-amz-content-sha256=[e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855], X-Amz-Date=[20200721T160746Z],
Authorization=[AWS4-HMAC-SHA256 Credential=..., SignedHeaders=content-md5;content-type;host;x-amz-content-sha256;x-amz-date,
Signature=fdb6a51c9cfbe116f8da033e079d4a9033a63b1d7a3abe8256546b2363f2dddb]}
> PUT >> https://my-temp-object-locked-bucket.s3-eu-central-1.amazonaws.com/multipart-uploaded-blob?partNumber=1&uploadId=7SXM.6rCAh_zyPYoF1KPyc86niKrkvUqoke00mEbJrAj9pdmp8Cf0cK6uGbxVBjEJNXkO8JVC6Yb5aQKI30WP4eohjdfFQOtY888JlUm6eOFKWy8lsmNZ5fBgRcPCdMD
> {Content-Type=[application/unknown], Content-Length=[11], Host=[my-temp-object-locked-bucket.s3-eu-central-1.amazonaws.com],
x-amz-content-sha256=[9e8f1b13c357dd00b99dd1760be67a3fdeb3ed1e3605307e549ab74c483a43fa], X-Amz-Date=[20200721T160746Z],
Authorization=[AWS4-HMAC-SHA256 Credential=.., SignedHeaders=content-length;content-type;host;x-amz-content-sha256;x-amz-date,
Signature=9aec5a51bbffaa95321a054d8d6aad40b7c7d87d6967562592f8aaf51bb5e7aa]}
> DELETE >> https://my-temp-object-locked-bucket.s3-eu-central-1.amazonaws.com/multipart-uploaded-blob?uploadId=7SXM.6rCAh_zyPYoF1KPyc86niKrkvUqoke00mEbJrAj9pdmp8Cf0cK6uGbxVBjEJNXkO8JVC6Yb5aQKI30WP4eohjdfFQOtY888JlUm6eOFKWy8lsmNZ5fBgRcPCdMD
> {Host=[my-temp-object-locked-bucket.s3-eu-central-1.amazonaws.com], x-amz-content-sha256=[e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855],
X-Amz-Date=[20200721T160746Z], Authorization=[AWS4-HMAC-SHA256 Credential=..., SignedHeaders=host;x-amz-content-sha256;x-amz-date,
Signature=f2e53347d054d7c4a08eb1d62b1ff828d90135d0563d85ee8547626788dfd290]}
> {code}
> It looks like the blob's MD5 hash is added to the POST to initiate the multipart upload
but is missing from the part's PUT itself.
>  
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Mime
View raw message