libcloud-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Torf <m...@torf.cc>
Subject Re: Binary data in Python 3 using LocalStorageDriver
Date Fri, 28 Aug 2015 10:42:32 GMT
I've filed a pull request:

https://github.com/apache/libcloud/pull/568


On 22.07.2015 09:30, Torf wrote:
> I've filed an issue:
> 
> https://issues.apache.org/jira/browse/LIBCLOUD-725
> 
> 
> On 21.07.2015 13:40, Torf wrote:
>> Great, will do that.
>>
>>
>> On 21.07.2015 08:21, Tomaz Muraus wrote:
>>> It would be great if you can submit a pull request with a fix and test
>>> cases upstream :)
>>>
>>> https://libcloud.readthedocs.org/en/latest/development.html#contributing
>>>
>>> On Tue, Jul 21, 2015 at 2:11 PM, Torf <mail@torf.cc> wrote:
>>>
>>>> Hi everyone,
>>>>
>>>> Thanks for your input, Chris! So it seems to be indeed a problem with
>>>> LocalStorageDriver, not with my understanding :)
>>>>
>>>> What can I do to help getting this fixed in the official distribution?
>>>> I'd be happy to provide a patch, test cases, etc.
>>>>
>>>>
>>>> Regards,
>>>> Torf
>>>>
>>>>
>>>> On 21.07.2015 00:01, Chris Richards wrote:
>>>>> Torf,
>>>>>
>>>>> I stumbled across the same thing and modified my local repository copy.
>>>> I
>>>>> don't know if this was the correct way for libcloud proper, but it moved
>>>> me
>>>>> past the point where I was stuck. (There may be other places that need
>>>> it,
>>>>> but for the APIs I'm using they're sufficient.) You'll also need to do
"$
>>>>> pip3 install lockfile" to use the LocalStorage driver.
>>>>>
>>>>> diff -r a61cf0eec1a7 -r ad3deb3173ce libcloud/storage/drivers/local.py
>>>>> --- a/libcloud/storage/drivers/local.py Wed Jun 25 18:59:43 2014 -0500
>>>>> +++ b/libcloud/storage/drivers/local.py Wed Jun 25 19:03:33 2014 -0500
>>>>> @@ -406,7 +406,7 @@
>>>>>
>>>>>          path = self.get_object_cdn_url(obj)
>>>>>
>>>>> -        with open(path) as obj_file:
>>>>> +        with open(path, 'rb') as obj_file:
>>>>>              for data in read_in_chunks(obj_file, chunk_size=chunk_size):
>>>>>                  yield data
>>>>>
>>>>> @@ -490,7 +490,7 @@
>>>>>          self._make_path(base_path)
>>>>>
>>>>>          with LockLocalStorage(obj_path):
>>>>> -            obj_file = open(obj_path, 'w')
>>>>> +            obj_file = open(obj_path, 'wb')
>>>>>              for data in iterator:
>>>>>                  obj_file.write(data)
>>>>>
>>>>> As for the upload, here's how I call it:
>>>>>
>>>>>         obj = driver.upload_object_via_stream (
>>>>>             BytesIO (data), container, objname,
>>>>>             extra={'content_type': 'application/octet-stream'})
>>>>>
>>>>> Again, I don't know if this is libcloud proper or not; it works for me!
>>>>> (Ship it!)
>>>>>
>>>>> HTH,
>>>>> Chris
>>>>>
>>>>> On Mon, Jul 20, 2015 at 4:43 PM, Torf <mail@torf.cc> wrote:
>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> I'm trying to store binary data in Python 3 to a ``LocalStorageDriver``
>>>>>> instance. If I upload data via ``upload_object`` from a file on disk
and
>>>>>> then use ``download_object_as_stream`` I get a stream of bytes, as
>>>>>> expected. However, uploading a stream of bytes manually using
>>>>>> ``upload_object_via_stream`` fails:
>>>>>>
>>>>>>
>>>>>>>>> container.upload_object_via_stream(io.BytesIO(b'foobar'),
'key')
>>>>>> Traceback (most recent call last):
>>>>>>   File "<stdin>", line 1, in <module>
>>>>>>   File
>>>>>>
>>>>>>
>>>> "/home/torf/projects/coba/venv/lib/python3.4/site-packages/libcloud/storage/base.py",
>>>>>> line 157, in upload_object_via_stream
>>>>>>     iterator, self, object_name, extra=extra, **kwargs)
>>>>>>   File
>>>>>>
>>>>>>
>>>> "/home/torf/projects/coba/venv/lib/python3.4/site-packages/libcloud/storage/drivers/local.py",
>>>>>> line 497, in upload_object_via_stream
>>>>>>     obj_file.close()
>>>>>>   File
>>>>>>
>>>>>>
>>>> "/home/torf/projects/coba/venv/lib/python3.4/site-packages/libcloud/storage/drivers/local.py",
>>>>>> line 69, in __exit__
>>>>>>     raise value
>>>>>>   File
>>>>>>
>>>>>>
>>>> "/home/torf/projects/coba/venv/lib/python3.4/site-packages/libcloud/storage/drivers/local.py",
>>>>>> line 495, in upload_object_via_stream
>>>>>>     obj_file.write(data)
>>>>>> TypeError: must be str, not bytes
>>>>>>
>>>>>>
>>>>>> The problem seems to be that ``LocalStorageDriver`` opens the target
>>>>>> file for writing in text mode ('w') instead of binary mode ('wb').
>>>>>>
>>>>>> I couldn't find any information on whether keys and values are supposed
>>>>>> to be (Unicode) strings or bytes (or both), so I am not sure whether
>>>>>> this is a bug in ``LocalStorageDriver`` or a misunderstanding on
my
>>>> side.
>>>>>>
>>>>>> This is on Python 3.4 and LibCloud 0.17.0.
>>>>>>
>>>>>>
>>>>>> Regards,
>>>>>> Torf
>>>>>>
>>>>>
>>>>
>>>>
>>>
>>
>>
>>
> 
> 
> 


Mime
View raw message