jclouds-notifications mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Andrew Gaul (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (JCLOUDS-1472) Multipart upload for AzureBlob fails when using InputStream for files greater than 32MB
Date Fri, 04 Jan 2019 23:46:00 GMT

     [ https://issues.apache.org/jira/browse/JCLOUDS-1472?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Andrew Gaul updated JCLOUDS-1472:
---------------------------------
    Description: 
There are two things to mention.

1. When uploading a big file we use multipart for "supposed" efficiency but we found it only
works when payload is passed as s `File`. Any kind of InputStream fails with the message 

{noformat}
org.jclouds.http.HttpResponseException: Stream closed connecting to PUT https://__/__/myfile_100MB.bin-20181208-074010dic-4361f219-88c1-48cf-8e06-a9cd3035a05e?comp=block&blockid=AAAAAw%3D%3D
HTTP/1.1
	at org.jclouds.http.internal.BaseHttpCommandExecutorService.invoke(BaseHttpCommandExecutorService.java:120)
	at org.jclouds.rest.internal.InvokeHttpMethod.invoke(InvokeHttpMethod.java:91)
	at org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:74)
	at org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:45)
	at org.jclouds.rest.internal.DelegatesToInvocationFunction.handle(DelegatesToInvocationFunction.java:156)
	at org.jclouds.rest.internal.DelegatesToInvocationFunction.invoke(DelegatesToInvocationFunction.java:123)
	at com.sun.proxy.$Proxy44.putBlock(Unknown Source)
	at org.jclouds.azureblob.blobstore.AzureBlobStore.uploadMultipartPart(AzureBlobStore.java:446)
	at org.jclouds.blobstore.internal.BaseBlobStore$BlobUploader.call(BaseBlobStore.java:396)
	at org.jclouds.blobstore.internal.BaseBlobStore$BlobUploader.call(BaseBlobStore.java:383)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
	at java.lang.Thread.getStackTrace(Thread.java:1559)
	at org.jclouds.blobstore.internal.BaseBlobStore.putMultipartBlob(BaseBlobStore.java:367)
	at org.jclouds.blobstore.internal.BaseBlobStore.putMultipartBlob(BaseBlobStore.java:347)
	at org.jclouds.azureblob.blobstore.AzureBlobStore.putBlob(AzureBlobStore.java:238)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.google.inject.internal.DelegatingInvocationHandler.invoke(DelegatingInvocationHandler.java:37)
	at com.sun.proxy.$Proxy45.putBlob(Unknown Source)
	at org.abelsromero.jclouds.App.putContent(App.java:86)
	at org.abelsromero.jclouds.App.testFile(App.java:47)
	at org.abelsromero.jclouds.App.doStuff(App.java:34)
	at org.abelsromero.jclouds.App.main(App.java:21)
{noformat}

Here is a repo to reproduce the issue: https://github.com/abelsromero/jclouds-azure-streams.
Beware that download is 170 MB in files.
To run it, create a file called "config.properties" with this form under {{src/main/resources}}

{noformat}
storage.account.name=
storage.account.key=
storage.container.name=
{noformat}

To enable and disable multipart, change the value here https://github.com/abelsromero/jclouds-azure-streams/blob/master/src/main/java/org/abelsromero/jclouds/App.java#L33

Right now we are considering disabling multipart on some scenarios and in other we are forced
to create temporal files, so none of this are really good.

2. Setting `jclouds.mpu.parts.size` to a different value than the default 32MB does not seem
to do anything.

  was:
There are two things to mention.

1. When uploading a big file we use multipart for "supposed" efficiency but we found it only
works when payload is passed as s `File`. Any kind of InputStream fails with the message 
```
org.jclouds.http.HttpResponseException: Stream closed connecting to PUT https://__/__/myfile_100MB.bin-20181208-074010dic-4361f219-88c1-48cf-8e06-a9cd3035a05e?comp=block&blockid=AAAAAw%3D%3D
HTTP/1.1
	at org.jclouds.http.internal.BaseHttpCommandExecutorService.invoke(BaseHttpCommandExecutorService.java:120)
	at org.jclouds.rest.internal.InvokeHttpMethod.invoke(InvokeHttpMethod.java:91)
	at org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:74)
	at org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:45)
	at org.jclouds.rest.internal.DelegatesToInvocationFunction.handle(DelegatesToInvocationFunction.java:156)
	at org.jclouds.rest.internal.DelegatesToInvocationFunction.invoke(DelegatesToInvocationFunction.java:123)
	at com.sun.proxy.$Proxy44.putBlock(Unknown Source)
	at org.jclouds.azureblob.blobstore.AzureBlobStore.uploadMultipartPart(AzureBlobStore.java:446)
	at org.jclouds.blobstore.internal.BaseBlobStore$BlobUploader.call(BaseBlobStore.java:396)
	at org.jclouds.blobstore.internal.BaseBlobStore$BlobUploader.call(BaseBlobStore.java:383)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
	at java.lang.Thread.getStackTrace(Thread.java:1559)
	at org.jclouds.blobstore.internal.BaseBlobStore.putMultipartBlob(BaseBlobStore.java:367)
	at org.jclouds.blobstore.internal.BaseBlobStore.putMultipartBlob(BaseBlobStore.java:347)
	at org.jclouds.azureblob.blobstore.AzureBlobStore.putBlob(AzureBlobStore.java:238)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.google.inject.internal.DelegatingInvocationHandler.invoke(DelegatingInvocationHandler.java:37)
	at com.sun.proxy.$Proxy45.putBlob(Unknown Source)
	at org.abelsromero.jclouds.App.putContent(App.java:86)
	at org.abelsromero.jclouds.App.testFile(App.java:47)
	at org.abelsromero.jclouds.App.doStuff(App.java:34)
	at org.abelsromero.jclouds.App.main(App.java:21)
```

Here is a repo to reproduce the issue: https://github.com/abelsromero/jclouds-azure-streams.
Beware that download is 170 MB in files.
To run it, create a file called "config.properties" with this form under {{src/main/resources}}

```
storage.account.name=
storage.account.key=
storage.container.name=
```

To enable and disable multipart, change the value here https://github.com/abelsromero/jclouds-azure-streams/blob/master/src/main/java/org/abelsromero/jclouds/App.java#L33

Right now we are considering disabling multipart on some scenarios and in other we are forced
to create temporal files, so none of this are really good.

2. Setting `jclouds.mpu.parts.size` to a different value than the default 32MB does not seem
to do anything.


> Multipart upload for AzureBlob fails when using InputStream for files greater than 32MB
> ---------------------------------------------------------------------------------------
>
>                 Key: JCLOUDS-1472
>                 URL: https://issues.apache.org/jira/browse/JCLOUDS-1472
>             Project: jclouds
>          Issue Type: Bug
>          Components: jclouds-blobstore
>    Affects Versions: 2.1.0, 2.1.1
>         Environment: Java 8 (191)
> Windows 10
>            Reporter: Abel Salgado Romero
>            Assignee: Andrew Gaul
>            Priority: Minor
>             Fix For: 2.2.0, 2.1.2
>
>
> There are two things to mention.
> 1. When uploading a big file we use multipart for "supposed" efficiency but we found
it only works when payload is passed as s `File`. Any kind of InputStream fails with the message

> {noformat}
> org.jclouds.http.HttpResponseException: Stream closed connecting to PUT https://__/__/myfile_100MB.bin-20181208-074010dic-4361f219-88c1-48cf-8e06-a9cd3035a05e?comp=block&blockid=AAAAAw%3D%3D
HTTP/1.1
> 	at org.jclouds.http.internal.BaseHttpCommandExecutorService.invoke(BaseHttpCommandExecutorService.java:120)
> 	at org.jclouds.rest.internal.InvokeHttpMethod.invoke(InvokeHttpMethod.java:91)
> 	at org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:74)
> 	at org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:45)
> 	at org.jclouds.rest.internal.DelegatesToInvocationFunction.handle(DelegatesToInvocationFunction.java:156)
> 	at org.jclouds.rest.internal.DelegatesToInvocationFunction.invoke(DelegatesToInvocationFunction.java:123)
> 	at com.sun.proxy.$Proxy44.putBlock(Unknown Source)
> 	at org.jclouds.azureblob.blobstore.AzureBlobStore.uploadMultipartPart(AzureBlobStore.java:446)
> 	at org.jclouds.blobstore.internal.BaseBlobStore$BlobUploader.call(BaseBlobStore.java:396)
> 	at org.jclouds.blobstore.internal.BaseBlobStore$BlobUploader.call(BaseBlobStore.java:383)
> 	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> 	at java.lang.Thread.run(Thread.java:748)
> 	at java.lang.Thread.getStackTrace(Thread.java:1559)
> 	at org.jclouds.blobstore.internal.BaseBlobStore.putMultipartBlob(BaseBlobStore.java:367)
> 	at org.jclouds.blobstore.internal.BaseBlobStore.putMultipartBlob(BaseBlobStore.java:347)
> 	at org.jclouds.azureblob.blobstore.AzureBlobStore.putBlob(AzureBlobStore.java:238)
> 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 	at java.lang.reflect.Method.invoke(Method.java:498)
> 	at com.google.inject.internal.DelegatingInvocationHandler.invoke(DelegatingInvocationHandler.java:37)
> 	at com.sun.proxy.$Proxy45.putBlob(Unknown Source)
> 	at org.abelsromero.jclouds.App.putContent(App.java:86)
> 	at org.abelsromero.jclouds.App.testFile(App.java:47)
> 	at org.abelsromero.jclouds.App.doStuff(App.java:34)
> 	at org.abelsromero.jclouds.App.main(App.java:21)
> {noformat}
> Here is a repo to reproduce the issue: https://github.com/abelsromero/jclouds-azure-streams.
Beware that download is 170 MB in files.
> To run it, create a file called "config.properties" with this form under {{src/main/resources}}
> {noformat}
> storage.account.name=
> storage.account.key=
> storage.container.name=
> {noformat}
> To enable and disable multipart, change the value here https://github.com/abelsromero/jclouds-azure-streams/blob/master/src/main/java/org/abelsromero/jclouds/App.java#L33
> Right now we are considering disabling multipart on some scenarios and in other we are
forced to create temporal files, so none of this are really good.
> 2. Setting `jclouds.mpu.parts.size` to a different value than the default 32MB does not
seem to do anything.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message