libcloud-notifications mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From to...@apache.org
Subject svn commit: r1342849 - in /libcloud/site/trunk/content/docs: storage-examples.mdtext using-libcloud-in-multi-threaded-and-async-environments.mdtext
Date Sat, 26 May 2012 04:10:23 GMT
Author: tomaz
Date: Sat May 26 04:10:23 2012
New Revision: 1342849

URL: http://svn.apache.org/viewvc?rev=1342849&view=rev
Log:
Update "Using Libcloud in async and multi threaded environments" page and add
"Efficiently download multiple files using gevent" example.

Modified:
    libcloud/site/trunk/content/docs/storage-examples.mdtext
    libcloud/site/trunk/content/docs/using-libcloud-in-multi-threaded-and-async-environments.mdtext

Modified: libcloud/site/trunk/content/docs/storage-examples.mdtext
URL: http://svn.apache.org/viewvc/libcloud/site/trunk/content/docs/storage-examples.mdtext?rev=1342849&r1=1342848&r2=1342849&view=diff
==============================================================================
--- libcloud/site/trunk/content/docs/storage-examples.mdtext (original)
+++ libcloud/site/trunk/content/docs/storage-examples.mdtext Sat May 26 04:10:23 2012
@@ -3,6 +3,7 @@ title: Storage -> Examples
 ## Examples
 
 * [Create a backup of a directory and directly stream it to CloudFiles](#example-1-streaming-upload)
+* [Efficiently download multiple files using gevent](#efficiently-download-multiple-files-using-gevent)
 
 <h3 id="example-1-streaming-upload">Create a backup of a directory and directly stream
it to CloudFiles</h3>
 
@@ -39,3 +40,44 @@ title: Storage -> Examples
         return_code = pipe.poll()
 
     print 'Upload complete, transferred: %s KB' % ((obj.size / 1024))
+
+<h3 id="efficiently-download-multiple-files-using-gevent">Efficiently download multiple
files using gevent</h3>
+
+    ::python
+    import os.path
+
+    from gevent import monkey
+    from gevent.pool import Pool
+    monkey.patch_all()
+
+    from libcloud.storage.providers import get_driver
+    from libcloud.storage.types import Provider
+
+    USERNAME = 'username'
+    API_KEY = 'api key'
+
+    cls = get_driver(Provider.CLOUDFILES_US)
+    driver = cls(USERNAME, API_KEY)
+
+    def download_obj(container, obj):
+        driver = cls(USERNAME, API_KEY)
+        obj = driver.get_object(container_name=container.name,
+                                object_name=obj.name)
+        filename = os.path.basename(obj.name)
+        path = os.path.join(os.path.expanduser('~/Downloads'), filename)
+        print 'Downloading: %s to %s' % (obj.name, path)
+        obj.download(destination_path=path)
+
+    containers = driver.list_containers()
+
+    jobs = []
+    pool = Pool(20)
+
+    for index, container in enumerate(containers):
+        objects = container.list_objects()
+
+        for obj in objects:
+            pool.spawn(download_obj, container, obj)
+
+    pool.join()
+    print 'Done'

Modified: libcloud/site/trunk/content/docs/using-libcloud-in-multi-threaded-and-async-environments.mdtext
URL: http://svn.apache.org/viewvc/libcloud/site/trunk/content/docs/using-libcloud-in-multi-threaded-and-async-environments.mdtext?rev=1342849&r1=1342848&r2=1342849&view=diff
==============================================================================
--- libcloud/site/trunk/content/docs/using-libcloud-in-multi-threaded-and-async-environments.mdtext
(original)
+++ libcloud/site/trunk/content/docs/using-libcloud-in-multi-threaded-and-async-environments.mdtext
Sat May 26 04:10:23 2012
@@ -26,15 +26,21 @@ easiest solution is to create a new driv
 ### Using Libcloud with gevent ###
 
 gevent has an ability to monkey patch and replace functions in the Python `socket`
-and `time` module with its own functions which don't block.
+`urllib2`, `httplib` and `time` module with its own functions which don't block.
 
-The only thing you need to use to make Libcloud efficiently work with gevent is
-to enable this monkey-patching:
+You need to do two things when you want to use Libcloud with gevent:
+
+1. Enable monkey patching
 
     ::python
     from gevent import monkey
     monkey.patch_all()
 
+2. Create a separate driver instance for each Greenlet. This is necessary
+because a driver instance reuses the same `Connection` class.
+
+For example see [Efficiently download multiple files using gevent](/docs/storage-examples.html#efficiently-download-multiple-files-using-gevent).
+
 ### Using Libcloud with Twisted ###
 
 Libcloud has no Twisted support included in the core which means you need



Mime
View raw message