libcloud-notifications mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From build...@apache.org
Subject svn commit: r819146 - in /websites/staging/libcloud/trunk/content: ./ docs/storage-examples.html docs/using-libcloud-in-multi-threaded-and-async-environments.html
Date Sat, 26 May 2012 04:10:33 GMT
Author: buildbot
Date: Sat May 26 04:10:32 2012
New Revision: 819146

Log:
Staging update by buildbot for libcloud

Modified:
    websites/staging/libcloud/trunk/content/   (props changed)
    websites/staging/libcloud/trunk/content/docs/storage-examples.html
    websites/staging/libcloud/trunk/content/docs/using-libcloud-in-multi-threaded-and-async-environments.html

Propchange: websites/staging/libcloud/trunk/content/
------------------------------------------------------------------------------
--- cms:source-revision (original)
+++ cms:source-revision Sat May 26 04:10:32 2012
@@ -1 +1 @@
-1342848
+1342849

Modified: websites/staging/libcloud/trunk/content/docs/storage-examples.html
==============================================================================
--- websites/staging/libcloud/trunk/content/docs/storage-examples.html (original)
+++ websites/staging/libcloud/trunk/content/docs/storage-examples.html Sat May 26 04:10:32
2012
@@ -96,6 +96,7 @@
       <h2 id="examples">Examples</h2>
 <ul>
 <li><a href="#example-1-streaming-upload">Create a backup of a directory and
directly stream it to CloudFiles</a></li>
+<li><a href="#efficiently-download-multiple-files-using-gevent">Efficiently download
multiple files using gevent</a></li>
 </ul>
 <h3 id="example-1-streaming-upload">Create a backup of a directory and directly stream
it to CloudFiles</h3>
 
@@ -132,6 +133,48 @@
 
 <span class="k">print</span> <span class="s">&#39;Upload complete,
transferred: </span><span class="si">%s</span><span class="s"> KB&#39;</span>
<span class="o">%</span> <span class="p">((</span><span class="n">obj</span><span
class="o">.</span><span class="n">size</span> <span class="o">/</span>
<span class="mi">1024</span><span class="p">))</span>
 </pre></div>
+
+
+<h3 id="efficiently-download-multiple-files-using-gevent">Efficiently download multiple
files using gevent</h3>
+
+<div class="codehilite"><pre><span class="kn">import</span> <span
class="nn">os.path</span>
+
+<span class="kn">from</span> <span class="nn">gevent</span> <span
class="kn">import</span> <span class="n">monkey</span>
+<span class="kn">from</span> <span class="nn">gevent.pool</span>
<span class="kn">import</span> <span class="n">Pool</span>
+<span class="n">monkey</span><span class="o">.</span><span class="n">patch_all</span><span
class="p">()</span>
+
+<span class="kn">from</span> <span class="nn">libcloud.storage.providers</span>
<span class="kn">import</span> <span class="n">get_driver</span>
+<span class="kn">from</span> <span class="nn">libcloud.storage.types</span>
<span class="kn">import</span> <span class="n">Provider</span>
+
+<span class="n">USERNAME</span> <span class="o">=</span> <span
class="s">&#39;username&#39;</span>
+<span class="n">API_KEY</span> <span class="o">=</span> <span
class="s">&#39;api key&#39;</span>
+
+<span class="n">cls</span> <span class="o">=</span> <span class="n">get_driver</span><span
class="p">(</span><span class="n">Provider</span><span class="o">.</span><span
class="n">CLOUDFILES_US</span><span class="p">)</span>
+<span class="n">driver</span> <span class="o">=</span> <span class="n">cls</span><span
class="p">(</span><span class="n">USERNAME</span><span class="p">,</span>
<span class="n">API_KEY</span><span class="p">)</span>
+
+<span class="k">def</span> <span class="nf">download_obj</span><span
class="p">(</span><span class="n">container</span><span class="p">,</span>
<span class="n">obj</span><span class="p">):</span>
+    <span class="n">driver</span> <span class="o">=</span> <span
class="n">cls</span><span class="p">(</span><span class="n">USERNAME</span><span
class="p">,</span> <span class="n">API_KEY</span><span class="p">)</span>
+    <span class="n">obj</span> <span class="o">=</span> <span
class="n">driver</span><span class="o">.</span><span class="n">get_object</span><span
class="p">(</span><span class="n">container_name</span><span class="o">=</span><span
class="n">container</span><span class="o">.</span><span class="n">name</span><span
class="p">,</span>
+                            <span class="n">object_name</span><span class="o">=</span><span
class="n">obj</span><span class="o">.</span><span class="n">name</span><span
class="p">)</span>
+    <span class="n">filename</span> <span class="o">=</span> <span
class="n">os</span><span class="o">.</span><span class="n">path</span><span
class="o">.</span><span class="n">basename</span><span class="p">(</span><span
class="n">obj</span><span class="o">.</span><span class="n">name</span><span
class="p">)</span>
+    <span class="n">path</span> <span class="o">=</span> <span
class="n">os</span><span class="o">.</span><span class="n">path</span><span
class="o">.</span><span class="n">join</span><span class="p">(</span><span
class="n">os</span><span class="o">.</span><span class="n">path</span><span
class="o">.</span><span class="n">expanduser</span><span class="p">(</span><span
class="s">&#39;~/Downloads&#39;</span><span class="p">),</span>
<span class="n">filename</span><span class="p">)</span>
+    <span class="k">print</span> <span class="s">&#39;Downloading:
</span><span class="si">%s</span><span class="s"> to </span><span
class="si">%s</span><span class="s">&#39;</span> <span class="o">%</span>
<span class="p">(</span><span class="n">obj</span><span class="o">.</span><span
class="n">name</span><span class="p">,</span> <span class="n">path</span><span
class="p">)</span>
+    <span class="n">obj</span><span class="o">.</span><span class="n">download</span><span
class="p">(</span><span class="n">destination_path</span><span class="o">=</span><span
class="n">path</span><span class="p">)</span>
+
+<span class="n">containers</span> <span class="o">=</span> <span
class="n">driver</span><span class="o">.</span><span class="n">list_containers</span><span
class="p">()</span>
+
+<span class="n">jobs</span> <span class="o">=</span> <span class="p">[]</span>
+<span class="n">pool</span> <span class="o">=</span> <span class="n">Pool</span><span
class="p">(</span><span class="mi">20</span><span class="p">)</span>
+
+<span class="k">for</span> <span class="n">index</span><span class="p">,</span>
<span class="n">container</span> <span class="ow">in</span> <span
class="nb">enumerate</span><span class="p">(</span><span class="n">containers</span><span
class="p">):</span>
+    <span class="n">objects</span> <span class="o">=</span> <span
class="n">container</span><span class="o">.</span><span class="n">list_objects</span><span
class="p">()</span>
+
+    <span class="k">for</span> <span class="n">obj</span> <span
class="ow">in</span> <span class="n">objects</span><span class="p">:</span>
+        <span class="n">pool</span><span class="o">.</span><span
class="n">spawn</span><span class="p">(</span><span class="n">download_obj</span><span
class="p">,</span> <span class="n">container</span><span class="p">,</span>
<span class="n">obj</span><span class="p">)</span>
+
+<span class="n">pool</span><span class="o">.</span><span class="n">join</span><span
class="p">()</span>
+<span class="k">print</span> <span class="s">&#39;Done&#39;</span>
+</pre></div>
       
     </div>
 

Modified: websites/staging/libcloud/trunk/content/docs/using-libcloud-in-multi-threaded-and-async-environments.html
==============================================================================
--- websites/staging/libcloud/trunk/content/docs/using-libcloud-in-multi-threaded-and-async-environments.html
(original)
+++ websites/staging/libcloud/trunk/content/docs/using-libcloud-in-multi-threaded-and-async-environments.html
Sat May 26 04:10:32 2012
@@ -111,14 +111,21 @@ you don't want to deal with complex (and
 easiest solution is to create a new driver instance inside each thread.</p>
 <h3 id="using-libcloud-with-gevent">Using Libcloud with gevent</h3>
 <p>gevent has an ability to monkey patch and replace functions in the Python <code>socket</code>
-and <code>time</code> module with its own functions which don't block.</p>
-<p>The only thing you need to use to make Libcloud efficiently work with gevent is
-to enable this monkey-patching:</p>
-<div class="codehilite"><pre><span class="kn">from</span> <span
class="nn">gevent</span> <span class="kn">import</span> <span class="n">monkey</span>
-<span class="n">monkey</span><span class="o">.</span><span class="n">patch_all</span><span
class="p">()</span>
-</pre></div>
-
-
+<code>urllib2</code>, <code>httplib</code> and <code>time</code>
module with its own functions which don't block.</p>
+<p>You need to do two things when you want to use Libcloud with gevent:</p>
+<ol>
+<li>
+<p>Enable monkey patching</p>
+<p>::python
+from gevent import monkey
+monkey.patch_all()</p>
+</li>
+<li>
+<p>Create a separate driver instance for each Greenlet. This is necessary
+because a driver instance reuses the same <code>Connection</code> class.</p>
+</li>
+</ol>
+<p>For example see <a href="/docs/storage-examples.html#efficiently-download-multiple-files-using-gevent">Efficiently
download multiple files using gevent</a>.</p>
 <h3 id="using-libcloud-with-twisted">Using Libcloud with Twisted</h3>
 <p>Libcloud has no Twisted support included in the core which means you need
 to be careful when you use it with Twisted and some other async frameworks.</p>



Mime
View raw message