libcloud-notifications mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Joshua Hawkinson (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (LIBCLOUD-986) List / Iterate container with prefix for Azure Blob service (This support exists in the other drivers already)
Date Wed, 21 Mar 2018 17:52:00 GMT

    [ https://issues.apache.org/jira/browse/LIBCLOUD-986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16408315#comment-16408315
] 

Joshua Hawkinson edited comment on LIBCLOUD-986 at 3/21/18 5:51 PM:
--------------------------------------------------------------------

Okay, done... Here are the diffs for your review:
{code:java}
diff --git a/libcloud/storage/drivers/azure_blobs.py b/libcloud/storage/drivers/azure_blobs.py
index 82ee03a..2475793 100644
--- a/libcloud/storage/drivers/azure_blobs.py
+++ b/libcloud/storage/drivers/azure_blobs.py
@@ -381,11 +381,12 @@ class AzureBlobsStorageDriver(StorageDriver):
if not params['marker']:
break

-     def iterate_container_objects(self, container):
+     def iterate_container_objects(self, container, ex_prefix=None):
        """
        @inherits: :class:`StorageDriver.iterate_container_objects`
        """
-       params = {'restype': 'container',
+       params = {'prefix': ex_prefix,
+                 'restype': 'container',
                  'comp': 'list',
                  'maxresults': RESPONSES_PER_REQUEST,
                  'include': 'metadata'}
@@ -416,6 +417,22 @@ class AzureBlobsStorageDriver(StorageDriver):
           if not params['marker']:
               break

+    def list_container_objects(self, container, ex_prefix=None):
+      """
+      Return a list of objects for the given container.
+
+      :param container: Container instance.
+      :type container: :class:`Container`
+
+      :param ex_prefix: Only return objects starting with ex_prefix
+      :type ex_prefix: ``str``
+
+      :return: A list of Object instances.
+      :rtype: ``list`` of :class:`Object`
+      """
+      return list(self.iterate_container_objects(container,
+                                                 ex_prefix=ex_prefix))
+
def get_container(self, container_name):
"""
@inherits: :class:`StorageDriver.get_container`
diff --git a/libcloud/test/storage/test_azure_blobs.py b/libcloud/test/storage/test_azure_blobs.py
index 95ab3f1..69bdebb 100644
--- a/libcloud/test/storage/test_azure_blobs.py
+++ b/libcloud/test/storage/test_azure_blobs.py
@@ -447,6 +447,28 @@ class AzureBlobsTests(unittest.TestCase):
         self.assertTrue('content_encoding' in obj.extra)
         self.assertTrue('content_language' in obj.extra)

+    def test_list_container_objects_with_prefix(self):
+        self.mock_response_klass.type = None
+        AzureBlobsStorageDriver.RESPONSES_PER_REQUEST = 2
+
+        container = Container(name='test_container', extra={},
+                              driver=self.driver)
+        objects = self.driver.list_container_objects(container=container,
+        ex_prefix='test_prefix')
+        self.assertEqual(len(objects), 4)
+
+        obj = objects[1]
+        self.assertEqual(obj.name, 'object2.txt')
+        self.assertEqual(obj.hash, '0x8CFB90F1BA8CD8F')
+        self.assertEqual(obj.size, 1048576)
+        self.assertEqual(obj.container.name, 'test_container')
+        self.assertTrue('meta1' in obj.meta_data)
+        self.assertTrue('meta2' in obj.meta_data)
+        self.assertTrue('last_modified' in obj.extra)
+        self.assertTrue('content_type' in obj.extra)
+        self.assertTrue('content_encoding' in obj.extra)
+        self.assertTrue('content_language' in obj.extra)
+
{code}
 

And the before and after test results:
{code:java}
[root@localhost libcloud]# python3 -m pytest -k azure_blobs
=================================================================== test session starts ====================================================================
platform linux -- Python 3.4.3, pytest-3.4.2, py-1.5.2, pluggy-0.6.0
rootdir: /usr/libcloud, inifile: pytest.ini
collected 9116 items 

libcloud/test/storage/test_azure_blobs.py ....................F................ [100%]

========================================================================= FAILURES =========================================================================
_________________________________________________ AzureBlobsTests.test_list_container_objects_with_prefix
__________________________________________________

self = <libcloud.test.storage.test_azure_blobs.AzureBlobsTests testMethod=test_list_container_objects_with_prefix>

def test_list_container_objects_with_prefix(self):
self.mock_response_klass.type = None
AzureBlobsStorageDriver.RESPONSES_PER_REQUEST = 2

container = Container(name='test_container', extra={},
driver=self.driver)
objects = self.driver.list_container_objects(container=container,
> ex_prefix='test_prefix')
E TypeError: list_container_objects() got an unexpected keyword argument 'ex_prefix'

libcloud/test/storage/test_azure_blobs.py:457: TypeError
================================================================== 9079 tests deselected ===================================================================
=================================================== 1 failed, 36 passed, 9079 deselected in
3.39 seconds ===================================================
[root@localhost libcloud]# python3 -m pytest -k azure_blobs
=================================================================== test session starts ====================================================================
platform linux -- Python 3.4.3, pytest-3.4.2, py-1.5.2, pluggy-0.6.0
rootdir: /usr/libcloud, inifile: pytest.ini
collected 9116 items 

libcloud/test/storage/test_azure_blobs.py ..................................... [100%]

================================================================== 9079 tests deselected ===================================================================
======================================================== 37 passed, 9079 deselected in 3.08
seconds ========================================================{code}
Some notes about the unit test for this feature... Ideally you'd have one container that the
Mock HTTP server would be able to return different results for based on the prefix,  I didn't
see a way to simply accomplish this without modifying some of the base mock server code. 
This test method was largely pulled from the other driver's tests, but it seems to meet your
base requirements.  Another note,  I'm not sure why this Jira instance always strips spaces,
even in noformat, or code blocks (Sorry about the formatting).

As a bonus,  Here are my personal unit test results using my Azure account
{code:java}
LOG.info("Testing basic object store operations...")

LOG.info("Creating unit test bucket %s" % UNITTEST_BUCKET)
obj_stores.create_container(UNITTEST_BUCKET)

LOG.info("Getting bucket....")
bucket = obj_stores.get_container(UNITTEST_BUCKET)

LOG.info("Creating a couple of files")
_ = obj_stores.upload_object(__file__, container=bucket, object_name=UNITTEST_NAME1)
_ = obj_stores.upload_object(__file__, container=bucket, object_name=UNITTEST_NAME2)

LOG.info("Validating list directory like behavior")
obj_list = list(obj_stores.iterate_container_objects(bucket, ex_prefix=UNITTEST_DIR1))
obj_list2 = obj_stores.list_container_objects(bucket, ex_prefix=UNITTEST_DIR2)

{code}
 

output
{code:java}
2018-03-15 16:42:29 CAELUM_UNIT_TEST INFO 139630231111488 Starting unit test for object store
abstraction...
2018-03-15 16:42:29 CAELUM_UNIT_TEST INFO 139630231111488 ###### TEST VALIDATES CONFIG FILE
AND AWS CONNECTION
2018-03-15 16:42:29 CAELUM_UNIT_TEST INFO 139630231111488 Forcing config file read...
2018-03-15 16:42:29 CAELUM_UNIT_TEST INFO 139630231111488 Connecting to BUILDER AWS account
2018-03-15 16:42:29 CAELUM_UNIT_TEST INFO 139630231111488 Testing basic object store operations...
2018-03-15 16:42:29 CAELUM_UNIT_TEST INFO 139630231111488 Creating unit test bucket peaxy-caelum-unit-test
2018-03-15 16:42:30 CAELUM_UNIT_TEST INFO 139630231111488 Getting bucket....
2018-03-15 16:42:30 CAELUM_UNIT_TEST INFO 139630231111488 Creating a couple of files
2018-03-15 16:42:31 CAELUM_UNIT_TEST INFO 139630231111488 Validating list directory like behavior
2018-03-15 18:00:31 CAELUM_UNIT_TEST INFO 139630231111488 Attempting to delete bucket
2018-03-15 18:00:35 CAELUM_UNIT_TEST INFO 139630231111488 ###### TEST PASSED
2018-03-15 18:00:35 CAELUM_UNIT_TEST INFO 139630231111488 ###### TEST VALIDATES CONFIG VARIABLES
AND AZURE CONNECTION
2018-03-15 18:00:35 CAELUM_UNIT_TEST INFO 139630231111488 Instantiating object and then manually
setting the config variables
2018-03-15 18:00:35 CAELUM_UNIT_TEST INFO 139630231111488 Connecting to MSDN Azure account
2018-03-15 18:00:35 CAELUM_UNIT_TEST INFO 139630231111488 Testing basic object store operations...
2018-03-15 18:00:35 CAELUM_UNIT_TEST INFO 139630231111488 Creating unit test bucket peaxy-caelum-unit-test
2018-03-15 18:00:35 CAELUM_UNIT_TEST INFO 139630231111488 Getting bucket....
2018-03-15 18:00:35 CAELUM_UNIT_TEST INFO 139630231111488 Creating a couple of files
2018-03-15 18:00:35 CAELUM_UNIT_TEST INFO 139630231111488 Validating list directory like behavior

>>> obj_list
[<Object: name=testing1/caelum_test1, size=5709, hash=0x8D58AD95464B16D, provider=Microsoft
Azure (blobs) ...>]
>>> obj_list2
[<Object: name=testing2/caelum_test2, size=5709, hash=0x8D58AD954709A90, provider=Microsoft
Azure (blobs) ...>]{code}
 


was (Author: sehafoc):
Okay, done... Here are the diffs for your review:
{code:java}
diff --git a/libcloud/storage/drivers/azure_blobs.py b/libcloud/storage/drivers/azure_blobs.py
index 82ee03a..2475793 100644
--- a/libcloud/storage/drivers/azure_blobs.py
+++ b/libcloud/storage/drivers/azure_blobs.py
@@ -381,11 +381,12 @@ class AzureBlobsStorageDriver(StorageDriver):
if not params['marker']:
break

-     def iterate_container_objects(self, container):
+     def iterate_container_objects(self, container, ex_prefix=None):
        """
        @inherits: :class:`StorageDriver.iterate_container_objects`
        """
-       params = {'restype': 'container',
+       params = {'prefix': ex_prefix,
+                 'restype': 'container',
                  'comp': 'list',
                  'maxresults': RESPONSES_PER_REQUEST,
                  'include': 'metadata'}
@@ -416,6 +417,22 @@ class AzureBlobsStorageDriver(StorageDriver):
           if not params['marker']:
               break

+    def list_container_objects(self, container, ex_prefix=None):
+      """
+      Return a list of objects for the given container.
+
+      :param container: Container instance.
+      :type container: :class:`Container`
+
+      :param ex_prefix: Only return objects starting with ex_prefix
+      :type ex_prefix: ``str``
+
+      :return: A list of Object instances.
+      :rtype: ``list`` of :class:`Object`
+      """
+      return list(self.iterate_container_objects(container,
+                                                 ex_prefix=ex_prefix))
+
def get_container(self, container_name):
"""
@inherits: :class:`StorageDriver.get_container`
diff --git a/libcloud/test/storage/test_azure_blobs.py b/libcloud/test/storage/test_azure_blobs.py
index 95ab3f1..69bdebb 100644
--- a/libcloud/test/storage/test_azure_blobs.py
+++ b/libcloud/test/storage/test_azure_blobs.py
@@ -447,6 +447,28 @@ class AzureBlobsTests(unittest.TestCase):
         self.assertTrue('content_encoding' in obj.extra)
         self.assertTrue('content_language' in obj.extra)

+    def test_list_container_objects_with_prefix(self):
+        self.mock_response_klass.type = None
+        AzureBlobsStorageDriver.RESPONSES_PER_REQUEST = 2
+
+        container = Container(name='test_container', extra={},
+                              driver=self.driver)
+        objects = self.driver.list_container_objects(container=container,
+        ex_prefix='test_prefix')
+        self.assertEqual(len(objects), 4)
+
+        obj = objects[1]
+        self.assertEqual(obj.name, 'object2.txt')
+        self.assertEqual(obj.hash, '0x8CFB90F1BA8CD8F')
+        self.assertEqual(obj.size, 1048576)
+        self.assertEqual(obj.container.name, 'test_container')
+        self.assertTrue('meta1' in obj.meta_data)
+        self.assertTrue('meta2' in obj.meta_data)
+        self.assertTrue('last_modified' in obj.extra)
+        self.assertTrue('content_type' in obj.extra)
+        self.assertTrue('content_encoding' in obj.extra)
+        self.assertTrue('content_language' in obj.extra)
+
{code}
 

And the before and after test results:
{code:java}
[root@localhost libcloud]# python3 -m pytest -k azure_blobs
=================================================================== test session starts ====================================================================
platform linux -- Python 3.4.3, pytest-3.4.2, py-1.5.2, pluggy-0.6.0
rootdir: /usr/libcloud, inifile: pytest.ini
collected 9116 items 

libcloud/test/storage/test_azure_blobs.py ....................F................ [100%]

========================================================================= FAILURES =========================================================================
_________________________________________________ AzureBlobsTests.test_list_container_objects_with_prefix
__________________________________________________

self = <libcloud.test.storage.test_azure_blobs.AzureBlobsTests testMethod=test_list_container_objects_with_prefix>

def test_list_container_objects_with_prefix(self):
self.mock_response_klass.type = None
AzureBlobsStorageDriver.RESPONSES_PER_REQUEST = 2

container = Container(name='test_container', extra={},
driver=self.driver)
objects = self.driver.list_container_objects(container=container,
> ex_prefix='test_prefix')
E TypeError: list_container_objects() got an unexpected keyword argument 'ex_prefix'

libcloud/test/storage/test_azure_blobs.py:457: TypeError
================================================================== 9079 tests deselected ===================================================================
=================================================== 1 failed, 36 passed, 9079 deselected in
3.39 seconds ===================================================
[root@localhost libcloud]# python3 -m pytest -k azure_blobs
=================================================================== test session starts ====================================================================
platform linux -- Python 3.4.3, pytest-3.4.2, py-1.5.2, pluggy-0.6.0
rootdir: /usr/libcloud, inifile: pytest.ini
collected 9116 items 

libcloud/test/storage/test_azure_blobs.py ..................................... [100%]

================================================================== 9079 tests deselected ===================================================================
======================================================== 37 passed, 9079 deselected in 3.08
seconds ========================================================{code}
Some notes about the unit test for this feature... Ideally you'd have one container that the
Mock HTTP server would be able to return different results for based on the prefix,  I didn't
see a way to simply accomplish this without modifying some of the base mock server code. 
This test method was largely pulled from the other driver's tests, but it seems to meet your
base requirements.  Another note,  I'm not sure why this Jira instance always strips spaces,
even in noformat, or code blocks (Sorry about the formatting).

 

> List / Iterate container with prefix for Azure Blob service (This support exists in the
other drivers already)
> --------------------------------------------------------------------------------------------------------------
>
>                 Key: LIBCLOUD-986
>                 URL: https://issues.apache.org/jira/browse/LIBCLOUD-986
>             Project: Libcloud
>          Issue Type: Improvement
>          Components: Storage
>            Reporter: Joshua Hawkinson
>            Priority: Major
>
> While testing out the libcloud storage abstraction I've come across a curious omission
in driver support. It seems that for many drivers there is additional support for the {{list_container_objects()}} and
{{iterate_container_objects()}} to support a prefix (or {{ex_prefix)}}. 
> This functionality is not present in the Azure driver for some reason.   I've gone
and checked the list_blobs API and it also supports this functionality as seen here:
> [https://docs.microsoft.com/en-us/rest/api/storageservices/list-blobs]
>  
> I've attempted a quick and dirty patch that seems to work as well.
> {code}
> azure_blobs.py Line 384
> -    def iterate_container_objects(self, container):
> +    def iterate_container_objects(self, container, ex_prefix=None):
>         """
>         @inherits: :class:`StorageDriver.iterate_container_objects`
>         """
>         params = {'restype': 'container',
>                   'comp': 'list',
> +                  'prefix': ex_prefix,
>                   'maxresults': RESPONSES_PER_REQUEST,
>                   'include': 'metadata'}
> Line 419 (Add new function)
>     def list_container_objects(self, container, ex_prefix=None):
>         """
>         Return a list of objects for the given container.
>     
>         :param container: Container instance.
>         :type container: :class:`Container`
>     
>         :param ex_prefix: Only return objects starting with ex_prefix
>         :type ex_prefix: ``str``
>     
>         :return: A list of Object instances.
>         :rtype: ``list`` of :class:`Object`
>         """
>         return list(self.iterate_container_objects(container,
>                                                    ex_prefix=ex_prefix))
>     
>  {code}
>  
> I've just worked around this at a higher level for now. There isn't any urgency on my
end,  but it would be nice if the abstraction was more consistent so I could remove the workaround
code.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message