libcloud-notifications mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From GitBox <...@apache.org>
Subject [GitHub] [libcloud] c-w commented on a change in pull request #1287: [LIBCLOUD-1043] Fix Azure upload_object_via_stream used with iter
Date Sun, 26 May 2019 19:27:07 GMT
c-w commented on a change in pull request #1287: [LIBCLOUD-1043] Fix Azure upload_object_via_stream
used with iter
URL: https://github.com/apache/libcloud/pull/1287#discussion_r287611116
 
 

 ##########
 File path: libcloud/storage/drivers/azure_blobs.py
 ##########
 @@ -825,7 +826,12 @@ def upload_object_via_stream(self, iterator, container, object_name,
         """
         self._check_values(ex_blob_type, ex_page_blob_size)
         if ex_blob_type == "BlockBlob":
-            iterator.seek(0, os.SEEK_END)
+            try:
+                iterator.seek(0, os.SEEK_END)
+            except AttributeError:
+                buffer = BytesIO()
+                buffer.writelines(iterator)
 
 Review comment:
   Yes, this does buffer the whole iterator in memory unfortunately.
   
   Given that [the content-size header is required](https://docs.microsoft.com/en-us/rest/api/storageservices/put-blob#request-headers-all-blob-types)
we have to find the size of the iterator before making the request so I don't see a way to
avoid this for the general case. E.g. using [tee](https://docs.python.org/3/library/itertools.html#itertools.tee)
followed by something like [ilen](https://more-itertools.readthedocs.io/en/stable/_modules/more_itertools/more.html#ilen)
will still copy the iterator in memory.
   
   One potential work-around could be to upload chunks from the iterator via individual [put
block](https://docs.microsoft.com/en-us/rest/api/storageservices/put-block) requests followed
with a [put block list](https://docs.microsoft.com/en-us/rest/api/storageservices/put-block-list)
request, but that would be a much more invasive change to the codebase. (Note that I also
haven't proved out this approach in code yet so for now it's just a hypothesis from reading
the docs.)
   
   Do you have any suggestions to improve this?

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

Mime
View raw message