jclouds-notifications mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Timur Alperovich <notificati...@github.com>
Subject Re: [jclouds] JCLOUDS-894: Add portable multipart upload (#762)
Date Wed, 03 Jun 2015 22:36:33 GMT
> @@ -275,4 +285,20 @@ public String copyBlob(String fromContainer, String fromName, String
toContainer
>           Closeables2.closeQuietly(is);
>        }
>     }
> +
> +   // TODO: parallel uploads
> +   @Beta
> +   protected String putMultipartBlob(String container, Blob blob, PutOptions overrides)
{
> +      MultipartUpload mpu = initiateMultipartUpload(container, blob.getMetadata());
> +      List<MultipartPart> parts = Lists.newArrayList();
> +      long contentLength = blob.getMetadata().getContentMetadata().getContentLength();
> +      long partSize = getMaximumMultipartPartSize();  // TODO: optimal?

AWS Java S3 SDK does the following:

    public static long calculateOptimalPartSize(PutObjectRequest putObjectRequest, TransferManagerConfiguration
configuration) {
        double contentLength = TransferManagerUtils.getContentLength(putObjectRequest);
        double optimalPartSize = (double)contentLength / (double)MAXIMUM_UPLOAD_PARTS;
         // round up so we don't push the upload over the maximum number of parts
        optimalPartSize = Math.ceil(optimalPartSize);
        return (long)Math.max(optimalPartSize, configuration.getMinimumUploadPartSize());
    }

AWS SDK defaults to a maximum of 10000 parts. The minimum default part size is 5MB. So, uploading
a 51GB file, for example, would use 8500 6MB parts.

Jclouds could use a similar mechanism. It would probably make sense to expose the configuration
parameters to be able to change the default behavior.

---
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds/pull/762/files#r31677839
Mime
View raw message