libcloud-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Tomaz Muraus <to...@apache.org>
Subject Re: Binary data in Python 3 using LocalStorageDriver
Date Tue, 21 Jul 2015 06:21:04 GMT
It would be great if you can submit a pull request with a fix and test
cases upstream :)

https://libcloud.readthedocs.org/en/latest/development.html#contributing

On Tue, Jul 21, 2015 at 2:11 PM, Torf <mail@torf.cc> wrote:

> Hi everyone,
>
> Thanks for your input, Chris! So it seems to be indeed a problem with
> LocalStorageDriver, not with my understanding :)
>
> What can I do to help getting this fixed in the official distribution?
> I'd be happy to provide a patch, test cases, etc.
>
>
> Regards,
> Torf
>
>
> On 21.07.2015 00:01, Chris Richards wrote:
> > Torf,
> >
> > I stumbled across the same thing and modified my local repository copy.
> I
> > don't know if this was the correct way for libcloud proper, but it moved
> me
> > past the point where I was stuck. (There may be other places that need
> it,
> > but for the APIs I'm using they're sufficient.) You'll also need to do "$
> > pip3 install lockfile" to use the LocalStorage driver.
> >
> > diff -r a61cf0eec1a7 -r ad3deb3173ce libcloud/storage/drivers/local.py
> > --- a/libcloud/storage/drivers/local.py Wed Jun 25 18:59:43 2014 -0500
> > +++ b/libcloud/storage/drivers/local.py Wed Jun 25 19:03:33 2014 -0500
> > @@ -406,7 +406,7 @@
> >
> >          path = self.get_object_cdn_url(obj)
> >
> > -        with open(path) as obj_file:
> > +        with open(path, 'rb') as obj_file:
> >              for data in read_in_chunks(obj_file, chunk_size=chunk_size):
> >                  yield data
> >
> > @@ -490,7 +490,7 @@
> >          self._make_path(base_path)
> >
> >          with LockLocalStorage(obj_path):
> > -            obj_file = open(obj_path, 'w')
> > +            obj_file = open(obj_path, 'wb')
> >              for data in iterator:
> >                  obj_file.write(data)
> >
> > As for the upload, here's how I call it:
> >
> >         obj = driver.upload_object_via_stream (
> >             BytesIO (data), container, objname,
> >             extra={'content_type': 'application/octet-stream'})
> >
> > Again, I don't know if this is libcloud proper or not; it works for me!
> > (Ship it!)
> >
> > HTH,
> > Chris
> >
> > On Mon, Jul 20, 2015 at 4:43 PM, Torf <mail@torf.cc> wrote:
> >
> >> Hi,
> >>
> >> I'm trying to store binary data in Python 3 to a ``LocalStorageDriver``
> >> instance. If I upload data via ``upload_object`` from a file on disk and
> >> then use ``download_object_as_stream`` I get a stream of bytes, as
> >> expected. However, uploading a stream of bytes manually using
> >> ``upload_object_via_stream`` fails:
> >>
> >>
> >>>>> container.upload_object_via_stream(io.BytesIO(b'foobar'), 'key')
> >> Traceback (most recent call last):
> >>   File "<stdin>", line 1, in <module>
> >>   File
> >>
> >>
> "/home/torf/projects/coba/venv/lib/python3.4/site-packages/libcloud/storage/base.py",
> >> line 157, in upload_object_via_stream
> >>     iterator, self, object_name, extra=extra, **kwargs)
> >>   File
> >>
> >>
> "/home/torf/projects/coba/venv/lib/python3.4/site-packages/libcloud/storage/drivers/local.py",
> >> line 497, in upload_object_via_stream
> >>     obj_file.close()
> >>   File
> >>
> >>
> "/home/torf/projects/coba/venv/lib/python3.4/site-packages/libcloud/storage/drivers/local.py",
> >> line 69, in __exit__
> >>     raise value
> >>   File
> >>
> >>
> "/home/torf/projects/coba/venv/lib/python3.4/site-packages/libcloud/storage/drivers/local.py",
> >> line 495, in upload_object_via_stream
> >>     obj_file.write(data)
> >> TypeError: must be str, not bytes
> >>
> >>
> >> The problem seems to be that ``LocalStorageDriver`` opens the target
> >> file for writing in text mode ('w') instead of binary mode ('wb').
> >>
> >> I couldn't find any information on whether keys and values are supposed
> >> to be (Unicode) strings or bytes (or both), so I am not sure whether
> >> this is a bug in ``LocalStorageDriver`` or a misunderstanding on my
> side.
> >>
> >> This is on Python 3.4 and LibCloud 0.17.0.
> >>
> >>
> >> Regards,
> >> Torf
> >>
> >
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message