libcloud-notifications mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (LIBCLOUD-851) Azure blob storage - Unable to list blobs of a container
Date Mon, 16 Apr 2018 22:39:00 GMT

    [ https://issues.apache.org/jira/browse/LIBCLOUD-851?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16440126#comment-16440126
] 

ASF GitHub Bot commented on LIBCLOUD-851:
-----------------------------------------

GitHub user daviskirk opened a pull request:

    https://github.com/apache/libcloud/pull/1202

    Fix hash value in azure blob store

    
    ## Upgrade azure blob storage api
    
    ### Description
    
    * Upgrade to minimum api version allowed by azure (see )
    
    * Return actual file (md5) hash instead of etag ('"0xFFFFFFFFFFFFFFF"')
    
    The current trunk version of libcloud throws a Libcloud error also described in (https://issues.apache.org/jira/browse/LIBCLOUD-851)
    
    ```
    Traceback (most recent call last):
      File "...", line 12, in <module>
        c = d.list_containers()[1]
      File "libcloud/libcloud/storage/base.py", line 209, in list_containers
        return list(self.iterate_containers())
      File "libcloud/libcloud/storage/drivers/azure_blobs.py", line 371, in iterate_containers
        (response.status), driver=self)
    libcloud.common.types.LibcloudError: <LibcloudError in <libcloud.storage.drivers.azure_blobs.AzureBlobsStorageDriver
object at 0x7f7389bf1080> 'Unexpected status code: 400'>
    ```
    and the corresponing response body of the failing request is:
    ```
    response.body
    '<?xml version="1.0" encoding="utf-8"?><Error><Code>InvalidHeaderValue</Code><Message>The
value for one of the HTTP headers is not in the correct format.\nRequestId:dd262993-501e-0030-6cd1-d58682000000\nTime:2018-04-16T22:24:39.6566784Z</Message><HeaderName>x-ms-version</HeaderName><HeaderValue>2012-02-12</HeaderValue></Error>'
    ```
    
    From looking at:
    https://docs.microsoft.com/en-us/rest/api/storageservices/versioning-for-the-azure-storage-services#for-blob-storage-accounts
    
    the easiest fix is to upgrade the api version to the minumum supported version (2014-02-12).
    
    In addition, the other cloud drivers all provide the actual md5 hash when uploading or
getting an object.  This allows the user to check if a file has changed online compared to
locally.  The current master version provides the etag value as the "hash" attribute (interestingly
with literal quotes around the tag: `'"0xFFFFFFFFFFFFFFF"'`) which is not compatible with
the other drivers.
    
    Replace this with the PR description (mention the changes you have made, why
    you have made them, provide some background and any references to the provider
    documentation if needed, etc.).
    
    I am assuming that these are both bugs and not design decisions (if so feel free to close
this request).
    
    ### Status
    
    - work in progress
    
    ### Checklist (tick everything that applies)
    
    - [ ] [Code linting](http://libcloud.readthedocs.org/en/latest/development.html#code-style-guide)
(required, can be done after the PR checks)
    - [ ] Documentation
    - [ ] [Tests](http://libcloud.readthedocs.org/en/latest/testing.html)
    - [ ] [ICLA](http://libcloud.readthedocs.org/en/latest/development.html#contributing-bigger-changes)
(required for bigger changes)


You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/daviskirk/libcloud 851_fix-azure-blob-storage-api-version

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/libcloud/pull/1202.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #1202
    
----
commit c73beb6879353ff9689f23580a5ee3499782de05
Author: Davis Kirkendall <davis.e.kirkendall@...>
Date:   2018-04-16T22:05:02Z

    Fix hash value in azure blob store
    
    - Upgrade to minimum api version allowed by azure (see https://docs.microsoft.com/en-us/rest/api/storageservices/versioning-for-the-azure-storage-services#for-blob-storage-accounts)
    - Return actual file (md5) hash instead of etag ('"0xFFFFFFFFFFFFFFF"')

----


> Azure blob storage - Unable to list blobs of a container
> --------------------------------------------------------
>
>                 Key: LIBCLOUD-851
>                 URL: https://issues.apache.org/jira/browse/LIBCLOUD-851
>             Project: Libcloud
>          Issue Type: Bug
>          Components: Storage
>         Environment: Debian GNU/Linux
>            Reporter: Antoine R. Dumont
>            Priority: Major
>              Labels: blobstore, storage_drivers
>
> Hello,
> Trying to list the blobs of a container in an azure blob storage, the instanciation step
of a container systematically breaks with a 400 bad request.
> I have not much details in the error message:
> {code:title=output|borderStyle=solid}
> Traceback (most recent call last):
>   File "./bin/basic-scenario", line 20, in <module>
>     container = driver.get_container(container_name=container_name)
>   File "/usr/lib/python3/dist-packages/libcloud/storage/drivers/azure_blobs.py", line
441, in get_container
>     (response.status), driver=self)
> libcloud.common.types.LibcloudError: <LibcloudError in <libcloud.storage.drivers.azure_blobs.AzureBlobsStorageDriver
object at 0x7f674c6b67f0> 'Unexpected status code: 400'
> {code}
> Here is the code of ./bin/basic-scenario:
> {code:title=basic-scenario|language=python|borderStyle=solid}
> #!/usr/bin/env python3
> from libcloud.storage.types import Provider
> from libcloud.storage.providers import get_driver
> if __name__ == '__main__':
>    # setup
>    account_name = 'account-name'
>    key = 'secret-key'
>    container_name = 'container-name'
>    # libcloud setup
>    cls = get_driver(Provider.AZURE_BLOBS)
>    # this instanciates ok
>    driver = cls(key=account_name, secret=key)
>    # this explodes
>    container = driver.get_container(container_name=container_name)
>    for c in driver.iterate_container_objects(container):
>        print(c)
> {code}
> Trying to dig in the error, using LIBCLOUD_DEBUG:
> {code}
> LIBCLOUD_DEBUG=error-output ./bin/basic-scenario
> {code}
> I have some more input, here is the content of error-output file:
> {code}
> # -------- begin 140359483662744 request ----------
> curl -i --head -H 'Authorization: SharedKey account-name:blabla/blabla/blabla=' -H 'x-ms-version:
2012-02-12' -H 'User-Agent: libcloud/0.20.0 (Microsoft Azure (blobs)) ' -H 'Accept-Encoding:
gzip,deflate' -H 'x-ms-date: Fri, 09 Sep 2016 10:19:36 GMT' -H 'X-LC-Request-ID: 140340383662744'
--compress 'https://account-name.blob.core.windows.net:443/container-name?restype=container'
> # -------- begin 140359483662744:140359483745056 response ----------
> HTTP/1.1 400 The value for one of the HTTP headers is not in the correct format.
> Transfer-Encoding: chunked
> Server: Microsoft-HTTPAPI/2.0
> X-Ms-Request-Id: ad3456f8-0002-0076-2f83-0ab6c2888888
> Date: Fri, 09 Sep 2016 10:19:36 GMT
> 0
> 0
> # -------- end 140359483662744:140359483745056 response ----------
> {code}
> As my package is a debian one with what seems an older version (0.20.0, seen in CHANGES.rst
and not in the propose jira list), I tried with the current git repository (1.1.0).
> Procedure: 
> 1. purged python3-libcloud from my machine
> 2. `python3 setup.py install` from git repository.
> I have the same error:
> {code}
> # -------- begin 139921458559016 request ----------
> curl -i --head -H 'x-ms-version: 2012-02-12' -H 'Authorization: SharedKey account-name:blabloblu='
-H 'X-LC-Request-ID: 139921458559432' -H 'User-Agent: libcloud/1.1.0 (Microsoft Azure (blobs))
' -H 'Accept-Encoding: gzip,deflate' -H 'x-ms-date: Fri, 09 Sep 2016 10:51:53 GMT' --compress
'https://account-name.blob.core.windows.net:443/container-name?restype=container'
> # -------- begin 139921458559016:139921458672360 response ----------
> HTTP/1.1 400 The value for one of the HTTP headers is not in the correct format.
> Transfer-Encoding: chunked
> Server: Microsoft-HTTPAPI/2.0
> X-Ms-Request-Id: 10947125-0001-005e-7788-0ac27a888888
> Date: Fri, 09 Sep 2016 10:51:52 GMT
> 0
> 0
> # -------- end 139921458559016:139921458672360 response ----------
> {code}
> I'm stuck there since i don't see anything wrong with the headers.
> Do you please have some hint?
> Thanks for your time
> Note: 
> - I'm confident in the azure blob storage setup and credentials since i got it working
with another library (azure-storage via pip3)



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message