jclouds-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Veit Guna <veit.g...@gmx.de>
Subject Re: ApacheHC driver and stream closing
Date Wed, 11 Mar 2015 21:47:10 GMT
Hi Ignasi.

In fact, in the "patch" I provided for the other issue (S3 upload
performance), I already
removed the hashing lines and simply took the provided MD5 hash from the
ContentMetadata.
And that worked. But I wasn't sure about the intention of the additional
hashing,
so I thought I better ask for more insight here ;).

So no changes were needed for [1] and [2]. If it makes sense, I can
create a PR so someone
can review the changes.

Thanks,
Veit


Am 11.03.2015 um 22:23 schrieb Ignasi Barrera:
> Looking at the code I would say it is an issue, but at some point that
> should be intentional. The Apache HC driver explicitly excludes the
> Content-MD5 header when converting the jclouds request to its request
> object (see [1]).
>
> By looking at the other drivers (the default Java one and OkHttp),
> both use that method to populate the Content-MD5 header (here you can
> see what it does [2]) which does pretty much the same than what the
> ApacheHC does *after*, by computing the hash again.
>
> Could you try removing the lines that rehash the stream and populate
> the Content-MD5 header from the Apache HC driver, and add the
> Content-MD5 header to the list of "allowed headers" in the first link?
> I'm pretty sure that would work. And if that works, mind opening a
> pull request with the fix?
>
>
> HTH!
>
> I.
>
>
> [1] https://github.com/jclouds/jclouds/blob/master/drivers/apachehc/src/main/java/org/jclouds/http/apachehc/ApacheHCUtils.java#L155-L162
> [2] https://github.com/jclouds/jclouds/blob/master/core/src/main/java/org/jclouds/io/ContentMetadataCodec.java#L98-L99
>
> On 11 March 2015 at 17:22, Veit Guna <Veit.Guna@gmx.de> wrote:
>> Hi.
>>
>> I have a problem with the ApacheHC driver using jclouds 1.8.1. I'm using it to upload
files
>> to an S3 bucket. For the payload I'm using an InputStream that is not resettable
nor "rereadable".
>> When I try to send the file via:
>>
>>         PayloadBlobBuilder blobBuilder = blobStore.blobBuilder("test")
>>                 .payload(inputStream)
>>                 .contentDisposition("attachment; filename=test")
>>                 .contentType("application/octet-stream")
>>                 .contentMD5(hash)
>>                 .contentLength(file.length()
>>                         );
>>
>> It dies with the exception below.
>>
>> A quick look at ApacheHCHttpCommandExecutorService shows:
>>
>>
>>    @Override
>>    protected HttpUriRequest convert(HttpRequest request) throws IOException {
>>       HttpUriRequest returnVal = apacheHCUtils.convertToApacheRequest(request);
>>       if (request.getPayload() != null && request.getPayload().getContentMetadata().getContentMD5()
!= null) {
>>          String md5 = base64().encode(ByteStreams2.hashAndClose(request.getPayload().openStream(),
md5()).asBytes());
>>          returnVal.addHeader("Content-MD5", md5);
>>       }
>>       return returnVal;
>>    }
>>
>> That means, that if an MD5 is specified, it reads the whole(!) stream to create his
own MD5. Why is that? The "client" already
>> supplied an MD5. So why hashing it again?
>>
>> Does that mean, using Md5 together with ApacheHC and non "rereadable" streams isn't
working at all? Is that by design?
>> The normal JavaUrlHttp driver works like a charm though.
>>
>> Maybe someone could shed some light on this :)?
>>
>> Thanks!
>> Veit
>>
>>
>>
>> Exception in thread "main" org.jclouds.http.HttpResponseException: null connecting
to PUT https://mybucket.s3-eu-west-1.amazonaws.com/test HTTP/1.1
>>     at org.jclouds.http.internal.BaseHttpCommandExecutorService.invoke(BaseHttpCommandExecutorService.java:110)
>>     at org.jclouds.rest.internal.InvokeHttpMethod.invoke(InvokeHttpMethod.java:90)
>>     at org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:73)
>>     at org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:44)
>>     at org.jclouds.rest.internal.DelegatesToInvocationFunction.handle(DelegatesToInvocationFunction.java:156)
>>     at org.jclouds.rest.internal.DelegatesToInvocationFunction.invoke(DelegatesToInvocationFunction.java:123)
>>     at com.sun.proxy.$Proxy42.putObject(Unknown Source)
>>     at org.jclouds.s3.blobstore.S3BlobStore.putBlob(S3BlobStore.java:235)
>>     at org.jclouds.aws.s3.blobstore.AWSS3BlobStore.putBlob(AWSS3BlobStore.java:95)
>>     at org.jclouds.s3.blobstore.S3BlobStore.putBlob(S3BlobStore.java:213)
>>     at org.jclouds.test.S3UploadTest.performUpload(S3UploadTest.java:71)
>>     at org.jclouds.test.S3UploadTest.main(S3UploadTest.java:36)
>> Caused by: org.apache.http.client.ClientProtocolException
>>     at org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:867)
>>     at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:115)
>>     at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:57)
>>     at org.jclouds.http.apachehc.ApacheHCHttpCommandExecutorService.executeRequest(ApacheHCHttpCommandExecutorService.java:109)
>>     at org.jclouds.http.apachehc.ApacheHCHttpCommandExecutorService.invoke(ApacheHCHttpCommandExecutorService.java:78)
>>     at org.jclouds.http.apachehc.ApacheHCHttpCommandExecutorService.invoke(ApacheHCHttpCommandExecutorService.java:52)
>>     at org.jclouds.http.internal.BaseHttpCommandExecutorService.invoke(BaseHttpCommandExecutorService.java:89)
>>     ... 11 more
>> Caused by: org.apache.http.client.NonRepeatableRequestException: Cannot retry request
with a non-repeatable request entity.  The cause lists the reason the original request failed.
>>     at org.apache.http.impl.client.DefaultRequestDirector.tryExecute(DefaultRequestDirector.java:659)
>>     at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:487)
>>     at org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:863)
>>     ... 17 more
>> Caused by: java.io.IOException: Stream Closed
>>     at java.io.FileInputStream.readBytes(Native Method)
>>     at java.io.FileInputStream.read(FileInputStream.java:272)
>>     at org.apache.http.entity.InputStreamEntity.writeTo(InputStreamEntity.java:138)
>>     at org.apache.http.entity.HttpEntityWrapper.writeTo(HttpEntityWrapper.java:89)
>>     at org.apache.http.impl.client.EntityEnclosingRequestWrapper$EntityWrapper.writeTo(EntityEnclosingRequestWrapper.java:108)
>>     at org.apache.http.impl.entity.EntitySerializer.serialize(EntitySerializer.java:117)
>>     at org.apache.http.impl.AbstractHttpClientConnection.sendRequestEntity(AbstractHttpClientConnection.java:265)
>>     at org.apache.http.impl.conn.AbstractClientConnAdapter.sendRequestEntity(AbstractClientConnAdapter.java:231)
>>     at org.apache.http.protocol.HttpRequestExecutor.doSendRequest(HttpRequestExecutor.java:236)
>>     at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:121)
>>     at org.apache.http.impl.client.DefaultRequestDirector.tryExecute(DefaultRequestDirector.java:685)
>>     ... 19 more
>>
>>
>>
>>



Mime
View raw message