jclouds-notifications mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF subversion and git services (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (JCLOUDS-1368) GCS cannot upload multipart file larger than 1 GB
Date Fri, 09 Feb 2018 06:40:00 GMT

    [ https://issues.apache.org/jira/browse/JCLOUDS-1368?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16357998#comment-16357998
] 

ASF subversion and git services commented on JCLOUDS-1368:
----------------------------------------------------------

Commit 992e60d82dc979c26a668079adda614e234434df in jclouds's branch refs/heads/2.0.x from
[~gaul]
[ https://git-wip-us.apache.org/repos/asf?p=jclouds.git;h=992e60d ]

JCLOUDS-1368: Fix off-by-one in slicing algorithm


> GCS cannot upload multipart file larger than 1 GB
> -------------------------------------------------
>
>                 Key: JCLOUDS-1368
>                 URL: https://issues.apache.org/jira/browse/JCLOUDS-1368
>             Project: jclouds
>          Issue Type: Bug
>          Components: jclouds-blobstore
>    Affects Versions: 2.0.3
>            Reporter: Andrew Gaul
>            Assignee: Andrew Gaul
>            Priority: Major
>              Labels: google-cloud-storage
>             Fix For: 2.1.0, 2.0.4
>
>
> When uploading a file:
> {code:java}
> long size = 33L * (32L * 1024L * 1024L);
> ByteSource byteSource = TestUtils.randomByteSource().slice(0, size);
> blobStore.createContainerInLocation(null, containerName);
> Blob blob = blobStore.blobBuilder(blobName)
>       .payload(byteSource)
>       .contentLength(size)
>       .build();
> blobStore.putBlob(containerName, blob, PutOptions.Builder.multipart());
> {code}
> I see an error:
> {noformat}
> org.jclouds.http.HttpResponseException: command: POST https://www.googleapis.com/storage/v1/b/gaul-blobstore0/o/large-blob/compose
HTTP/1.1 failed with response: HTTP/1.1 400 Bad Request; content: [{
>  "error": {
>   "errors": [
>    {
>     "domain": "global",
>     "reason": "invalid",
>     "message": "The number of source components provided (33) exceeds the maximum (32)"
>    }
>   ],
>   "code": 400,
>   "message": "The number of source components provided (33) exceeds the maximum (32)"
>  }
> }
> ]
> {noformat}
> This is due to the wonky {{MultipartUploadSlicingAlgorithm}} slicing into (31) 32 MB
parts then creating a remainder 64 MB part that {{BaseBlobStore.putMultipartBlob}} does not
account for.  This interface seems stupid and better to create (32) 33 MB parts.  Originally
reported at:
> https://stackoverflow.com/questions/47048264/the-number-of-source-components-provided-78-exceeds-the-maximum-32



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message