jclouds-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Andrew Gaul <g...@apache.org>
Subject Re: S3 upload with unknown content length
Date Tue, 24 Mar 2015 11:48:25 GMT
On Tue, Mar 24, 2015 at 10:34:55AM +0100, Veit Guna wrote:
> It seems that in general it is somehow possible to upload files without knowing its content
length in advance like implemented here:
>  
> https://github.com/libin/s3distcp/blob/master/src/main/java/com/amazon/external/elasticmapreduce/s3distcp/MultipartUploadOutputStream.java
>  
> I'm wondering if such functionality is already implemented in jclouds or, if not, are
there're any plans
> to do so?
> 
> Using the jclouds MultiPartUpload still forces me to specify the content-length.

jclouds uses the multipart size to avoid buffering parts in-memory since
each requires an explicit size:

https://github.com/jclouds/jclouds/blob/master/apis/s3/src/main/java/org/jclouds/s3/blobstore/strategy/internal/SequentialMultipartUploadStrategy.java

The s3distcp implementation calculates these sizes by buffering each
part on-disk, incurring three times as much IO as the jclouds equivalent
(read and write to the temporary file).  Several users have asked for an
improved multipart interface although this is tricky to do efficiently.
Could you add your thoughts to:

https://issues.apache.org/jira/browse/JCLOUDS-769

-- 
Andrew Gaul
http://gaul.org/

Mime
View raw message