jclouds-notifications mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Alexander Chernavin (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (JCLOUDS-1487) Blob content length is mandatory while uploading to Azure
Date Wed, 16 Jan 2019 01:54:00 GMT

    [ https://issues.apache.org/jira/browse/JCLOUDS-1487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16743544#comment-16743544
] 

Alexander Chernavin commented on JCLOUDS-1487:
----------------------------------------------

[~gaul], here is what I found so far. If content length is set to -1 or if MD5 needs to be
calculated, then Azure SDK's CloudBlockBlob tries to read from the input stream to calculate
stream length and it's MD5. Then it resets the stream and uploads it using putBlob API.

However, there is some threshold in place here. If input stream length is more than 128Mb,
then putBlob REST API is skipped and some streaming approach is used. It's not clear yet
what exactly happens during this streaming and what kind of corresponding REST API is called
during this streaming. It might be related to [Block blobs explanation|https://docs.microsoft.com/en-us/rest/api/storageservices/understanding-block-blobs--append-blobs--and-page-blobs#about-block-blobs]:

_Storage clients default to a 128 MB maximum single blob upload, settable using the SingleBlobUploadThresholdInBytes
property of the BlobRequestOptions object. When a block blob upload is larger than the value
in this property, storage clients break the file into blocks._

> Blob content length is mandatory while uploading to Azure
> ---------------------------------------------------------
>
>                 Key: JCLOUDS-1487
>                 URL: https://issues.apache.org/jira/browse/JCLOUDS-1487
>             Project: jclouds
>          Issue Type: Bug
>          Components: jclouds-blobstore
>    Affects Versions: 2.1.1
>            Reporter: Alexander Chernavin
>            Priority: Major
>              Labels: azureblob
>         Attachments: length_exception.txt
>
>
> Content length is mandatory while uploading to Azure using JClouds portable Blob Store
API
> {code:java}
> Blob blob = blobStore.blobBuilder(blobName)
>  .payload(new InputStreamPayload(inputStream))
>  .contentType(contentType)
>  .build();
> blobStore.putBlob(container, blob);
> {code}
> [BindAzureBlobMetadataToRequest|https://github.com/fusesource/jclouds/blame/master/providers/azureblob/src/main/java/org/jclouds/azureblob/binders/BindAzureBlobMetadataToRequest.java] class
checks that content length is specified and is greater than zero in lines 58-59. However,
according to the Azure SDK documentation, Azure [CloudBlockBlob|https://azure.github.io/azure-sdk-for-java/com/microsoft/azure/storage/blob/CloudBlockBlob.html#upload-java.io.InputStream-long-]
also accepts -1 if content length is unknown: 
> {{length}} - A {{long}} which represents the length, in bytes, of the stream data,
or -1 if unknown.
>  
> We have a use case when we need to stream a big chunk of data of unknown size directly
into a cloud storage. Portable JClouds API allows to upload files into OpenStack SWIFT without
specifying their size, but it fails on Azure.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message