libcloud-notifications mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From anthonys...@apache.org
Subject svn commit: r1802206 - in /libcloud/site/trunk/source: _posts/2017-04-09-async-rfc.md _posts/2017-04-25-libcloud-2-0-0-released.md downloads.md index.html
Date Mon, 17 Jul 2017 20:03:07 GMT
Author: anthonyshaw
Date: Mon Jul 17 20:03:07 2017
New Revision: 1802206

URL: http://svn.apache.org/viewvc?rev=1802206&view=rev
Log:
update website

Added:
    libcloud/site/trunk/source/_posts/2017-04-09-async-rfc.md
    libcloud/site/trunk/source/_posts/2017-04-25-libcloud-2-0-0-released.md
Modified:
    libcloud/site/trunk/source/downloads.md
    libcloud/site/trunk/source/index.html

Added: libcloud/site/trunk/source/_posts/2017-04-09-async-rfc.md
URL: http://svn.apache.org/viewvc/libcloud/site/trunk/source/_posts/2017-04-09-async-rfc.md?rev=1802206&view=auto
==============================================================================
--- libcloud/site/trunk/source/_posts/2017-04-09-async-rfc.md (added)
+++ libcloud/site/trunk/source/_posts/2017-04-09-async-rfc.md Mon Jul 17 20:03:07 2017
@@ -0,0 +1,159 @@
+---
+layout: post
+title: Have your say - async support in Apache Libcloud
+author: Anthony Shaw
+tags:
+  - news
+  - tutorial
+---
+
+One of the big requests whilst we were replacing `httplib` with the `requests` package in
2.0 was why didn't
+we use a HTTP library that supports _asynchronous_ API calls.
+
+The intention for 2.0 and replacing the HTTP backend classes was to improve the usability
of the project, by making SSL
+certificates easier to manage, improving the maintainability of our source code by using
an active 3rd party package and
+also improving performance and stability.
+
+Apache Libcloud already has documentation on threaded libraries like gevent and callback-based
libraries like Twisted, see
+[using libcloud in multithreaded environments](https://libcloud.readthedocs.io/en/latest/other/using-libcloud-in-multithreaded-and-async-environments.html)
+for examples.
+
+[PEP 492](https://www.python.org/dev/peps/pep-0492/#), implemented in Python 3.5 provides
a new coroutine protocol using methods,
+`__await__` for classes, a coroutine method wrapper, or a method that returns a coroutine
object.
+Also async [iterators](https://www.python.org/dev/peps/pep-0492/#asynchronous-iterators-and-async-for)
and [context managers](https://www.python.org/dev/peps/pep-0492/#asynchronous-context-managers-and-async-with)
+have been introduced.
+
+We would like to take advantage of the new language features by offering APIs in Apache Libcloud
without breaking backward compatibility and
+compatibility for users of <Python 3.5.
+
+Use cases for this would be:
+
+- Being able to fetch `Node` or `StorageObject`s from multiple geographies or drivers simultaneously.
+- Being able to quickly upload or download storage objects by parallelizing operations on
the `StorageDriver`.
+- Being able to call a long-running API method (e.g. generate report), whilst running other
code.
+
+## Design 1 - async context managers [PR 1016](https://github.com/apache/libcloud/pull/1016)
+
+This design would allow drivers to operate in 2 modes, the first is for synchronous method
calls, they return list or object
+data as per usual. The second mode, API methods like `NodeDriver.list_nodes` would return
a [coroutine object](https://www.python.org/dev/peps/pep-0492/#coroutine-objects)
+and could be awaited or gathered using an event loop.
+
+```python
+import asyncio
+
+from integration.driver.test import TestNodeDriver
+from libcloud.async_util import AsyncSession
+
+driver = TestNodeDriver('apache', 'libcloud')
+
+async def run():
+    # regular API call
+    nodes = driver.list_nodes()
+
+    async with AsyncSession(driver) as async_instance:
+        nodes = await async_instance.list_nodes()
+
+    assert len(nodes) == 2
+
+loop = asyncio.get_event_loop()
+loop.run_until_complete(run())
+loop.close()
+```
+
+## Design 2 - Additional methods in each driver for coroutines [PR 1027](https://github.com/apache/libcloud/pull/1027)
+
+This is the second design concept for async support in Libcloud.
+
+The concept here is to have Asynchronous Mixins, `LibcloudConnection` uses requests and `LibcloudAsyncConnection`
uses aiohttp for async transport [see](https://github.com/tonybaloney/libcloud/blob/d4fe097476d2f02941e17d5e1b1d405fcf44c0f7/libcloud/connection_async.py#L22-L42)
+
+The LibcloudAsyncConnection is an implementation detail of AsyncConnection, which is the
API for the drivers to consume [see](https://github.com/tonybaloney/libcloud/blob/d4fe097476d2f02941e17d5e1b1d405fcf44c0f7/libcloud/common/base.py#L742-L778)
+
+The drivers then use this mixin for their custom connection classes, e.g.
+
+```python
+
+class GoogleStorageConnection(ConnectionUserAndKey, AsyncConnection):
+    ...
+
+```
+
+They then inherit from `libcloud.storage.base.StorageAsyncDriver`, which uses a new set of
base methods, e.g. `iterate_containers_async` and can be implemented like this:
+
+```python
+        async def iterate_containers_async(self):
+            response = await self.connection.request_async('/')
+            if response.status == httplib.OK:
+                containers = self._to_containers(obj=response.object,
+                                                 xpath='Buckets/Bucket')
+                return containers
+    
+            raise LibcloudError('Unexpected status code: %s' % (response.status),
+                                driver=self)
+```
+
+Now the consumer can more or less do this:
+
+```python
+from libcloud.storage.providers import get_driver
+from libcloud.storage.types import Provider
+
+import asyncio
+
+GoogleStorageDriver = get_driver(Provider.GOOGLE_STORAGE)
+driver = GoogleStorageDriver(key=KEY, secret=SECRET)
+
+def do_stuff_with_object(obj):
+    print(obj)
+
+async def run():
+    tasks = []
+    async for container in driver.iterate_containers_async():
+        async for obj in driver.iterate_container_objects_async(container):
+            tasks.append(asyncio.ensure_future(do_stuff_with_object(obj)))
+    await asyncio.gather(*tasks)
+
+loop = asyncio.get_event_loop()
+loop.run_until_complete(run())
+loop.close()
+```
+
+## Design 3 - Initializer with "*async*" mode
+
+This option is similar to 2, except that if a driver is instantiated with "`async=True`",
+then all driver class methods would return coroutine objects. Internally, it would
+patch the Connection class with the AsyncConnection class.
+
+The downside of this is that all method calls to a driver would need to be awaited or used
+by an event loop.
+
+```python
+from libcloud.storage.providers import get_driver
+from libcloud.storage.types import Provider
+
+import asyncio
+
+GoogleStorageDriver = get_driver(Provider.GOOGLE_STORAGE)
+driver = GoogleStorageDriver(key=KEY, secret=SECRET, async=True)
+
+def do_stuff_with_object(obj):
+    print(obj)
+
+async def run():
+    tasks = []
+    async for container in driver.iterate_containers():
+        async for obj in driver.iterate_container_objects(container):
+            tasks.append(asyncio.ensure_future(do_stuff_with_object(obj)))
+    await asyncio.gather(*tasks)
+
+loop = asyncio.get_event_loop()
+loop.run_until_complete(run())
+loop.close()
+```
+
+# Give us feedback
+
+Got a better idea? Have an API or design, the question we're asking is 
+"if you wanted to use Libcloud for an async application, what would the code look like?"
This helps us design
+the API and the implementation details can follow.
+
+Feel free to comment on the mailing list or on the pull requests, or raise your own pull-request
with an API design.
\ No newline at end of file

Added: libcloud/site/trunk/source/_posts/2017-04-25-libcloud-2-0-0-released.md
URL: http://svn.apache.org/viewvc/libcloud/site/trunk/source/_posts/2017-04-25-libcloud-2-0-0-released.md?rev=1802206&view=auto
==============================================================================
--- libcloud/site/trunk/source/_posts/2017-04-25-libcloud-2-0-0-released.md (added)
+++ libcloud/site/trunk/source/_posts/2017-04-25-libcloud-2-0-0-released.md Mon Jul 17 20:03:07
2017
@@ -0,0 +1,179 @@
+---
+layout: post
+title: Libcloud 2.0.0 released
+author: Anthony Shaw
+tags:
+  - news
+  - release announcement
+---
+
+We are pleased to announce the release of Libcloud 2.0.0.
+
+This major release brings many new features, improvements, bug-fixes, and drivers.
+
+## Release highlights
+
+* Apache Libcloud 2.0 series replaces the use of Python httplib with a hard dependency on
the `requests` package. Users no longer
+  have to specific Certificate Authority bundles when using Apache Libcloud
+* 10% performance improvement through the use of HTTP sessions
+* Support for buffered IO streams for storage drivers
+* Support for Python 3.6, deprecation of Python 3.2
+* 3 new drivers - [OnApp Compute](http://libcloud.readthedocs.io/en/latest/compute/drivers/onapp.html),
[OnApp DNS](http://libcloud.readthedocs.io/en/latest/dns/drivers/onapp.html), [1&1 Compute](http://libcloud.readthedocs.io/en/latest/compute/drivers/oneandone.html)
+* Lots of improvements to our Azure ARM support
+* Continuing udates to the Amazon drivers
+
+A detailed description of the 2.0 HTTP API is documented [here](http://libcloud.readthedocs.io/en/latest/other/changes_in_2_0.html)
+  
+### Compute API Changes
+
+- Outscale SAS doc improvements and logo update
+
+#### OnApp Changes
+
+- Add list images support for OnApp driver
+- Add keypair management to OnApp driver
+
+#### Amazon EC2 Changes
+
+- Add r4 instance types for AWS
+- Add support for AWS eu-west-2 and ca-central-1 regions
+- Add P2 GPU instance types
+- Add method to modify snapshot attribute for EC2
+- Add ENA support for EC2 compute images
+- Add support for forcing detachment of EBS volumes to EC2 driver
+- Add support for ModifyVolume and DescribeVolumesModifications
+- Added Import Snapshot and Describe Import Snapshot to EC2 compute driver
+- Add missing regions in AWS storage and compute drivers
+- Add SR-IOV net support to images in EC2 compute driver
+- Fix - update t2.small image size from 11 CPU to 1
+- Added Billing Product for image in EC2 compute driver
+
+#### Linode changes
+
+- Add start, stop instance methods and fix incorrect state TERMINATED to STOPPED
+
+#### Azure ARM changes
+
+- Fix typeerror on ex_list_nics
+- Add support for Azure Cloud Environments as well as Locations
+- Fix string representation of the VhdImage type and fix listing of Public IP addresses
+- Add network security groups to azure ARM
+- Add the ability to list resource groups
+- Fix Azure ARM driver condition for ex_list_publishers where location is specified
+
+#### Google Cloud changes
+
+- Allow delete instances from managed group
+- Allow preemptible instances to be created
+- Remove validation checks for guestOsFeatures
+
+#### Alibaba Aliyun changes
+
+- Add aliyun ecs instance join leave security group
+- Fix Aliyun ECS, Load balancer and storage adapters when using unicode UTF-8 characters
in the names of resources
+
+#### DigitalOcean changes
+
+- Add price_monthly extra param to digitalocean sizes
+
+#### VMWare vSphere changes
+
+- Fix issue with authentication methods crashing
+
+### Storage
+
+- Reintroduce S3 multipart upload support with signature v4
+
+### DNS
+
+- Add [OnApp driver](http://libcloud.readthedocs.io/en/latest/dns/drivers/onapp.html)
+
+## Minor changes
+
+### Common
+
+- Added an integration test API and a test suite for validating functionality
+  without mocking any libcloud subsystems
+- Change Cloudscale to cloudscale.ch.
+
+## Bug Fixes
+
+### Compute
+
+#### 2.0.0
+
+- Fix OpenStack drivers not correctly setting URLs when used with identity API, would default
to 127.0.0.1 and service
+  catalog URLs were not adhered to.
+
+- Fix Aliyun ECS, Load balancer and storage adapters when using unicode UTF-8 characters
in the names of resources
+  in 2.0.0rc2 < it would fail as a MalformedResponseError, Python 2.7 element tree was
raising a unicode error
+
+- Refactor the test classes to use the full libcloud.http and libcloud.common.base modules,
with Connection,
+  Response all used with requests_mock. This increases our test coverages and catches bugs
in drivers' custom
+  parse_body and auth modules
+
+- Rename libcloud.httplib_ssl to libcloud.http now that we don't use httplib
+
+#### 2.0.0rc2
+
+- Fix Public IP not assigned when creating NIC on Azure ARM
+- Fix a bug in profitbricks driver where listing snapshots would request a malformed URL
+- Fix LIBCLOUD-806 bug where vsphere driver cannot be instantiated
+- [google compute] Improve performance of list nodes by caching volume information.
+
+  
+Full change log can be found at [here][1].
+
+### Special thank you
+
+I would like to wish a special thank you to all of our community contributors
+for their ongoing support to the project.
+
+- Tinu Cleatus for the OnApp driver
+- Alex Misstear for the EC2 improvements
+- Jie Ren for the Aliyun improvements
+- Francisco Ros for the DigitalOcean improvements
+- Peter Amstutz and Joseph Hall for the Azure ARM API updates
+- All of our community for their ongoing support and contributions...
+
+### Download
+
+The release can can be downloaded from
+<https://libcloud.apache.org/downloads.html> or installed using pip:
+
+<pre>
+pip install apache-libcloud==2.0.0
+</pre>
+
+### Upgrading
+
+If you have installed Libcloud using pip you can also use it to upgrade it:
+
+<pre>
+pip install --upgrade apache-libcloud==2.0.0
+</pre>
+
+### Upgrade notes
+
+A page which describes backward incompatible or semi-incompatible
+changes and how to preserve the old behavior when this is possible
+can be found at <https://libcloud.readthedocs.org/en/latest/upgrade_notes.html>
+
+### Documentation
+
+Regular and API documentation is available at <https://libcloud.readthedocs.org/en/latest/>
+
+### Bugs / Issues
+
+If you find any bug or issue, please report it on our issue tracker
+<https://issues.apache.org/jira/browse/LIBCLOUD>.
+Don't forget to attach an example and / or test which reproduces your
+problem.
+
+### Thanks
+
+Thanks to everyone who contributed and made this release possible! Full
+list of people who contributed to this release can be found in the
+[CHANGES file][1].
+
+[1]: https://libcloud.readthedocs.org/en/latest/changelog.html

Modified: libcloud/site/trunk/source/downloads.md
URL: http://svn.apache.org/viewvc/libcloud/site/trunk/source/downloads.md?rev=1802206&r1=1802205&r2=1802206&view=diff
==============================================================================
--- libcloud/site/trunk/source/downloads.md (original)
+++ libcloud/site/trunk/source/downloads.md Mon Jul 17 20:03:07 2017
@@ -9,13 +9,13 @@ description: Download the latest version
 This page contains link to the latest stable release of Libcloud. If you are
 interested in an older release, please visit the [archive page][15].
 
-<a name="latest-stable-release" id="latest-stable-release"><h2 class="anchor">Libcloud
2.0.0rc2 (stable) - Released on April 7, 2017</h2></a>
+<a name="latest-stable-release" id="latest-stable-release"><h2 class="anchor">Libcloud
2.1.0 (stable) - Released on April 25, 2017</h2></a>
 
 Files:
 
-* [apache-libcloud-2.0.0rc2.tar.gz][2] [[asc][3]] [[sha1][4]] [[md5][5]]
-* [apache-libcloud-2.0.0rc2.tar.bz2][6] [[asc][7]] [[sha1][8]] [[md5][9]]
-* [apache-libcloud-2.0.0rc2.tar.zip][10] [[asc][11]] [[sha1][12]] [[md5][13]]
+* [apache-libcloud-2.1.0.tar.gz][2] [[asc][3]] [[sha1][4]] [[md5][5]]
+* [apache-libcloud-2.1.0.tar.bz2][6] [[asc][7]] [[sha1][8]] [[md5][9]]
+* [apache-libcloud-2.1.0.tar.zip][10] [[asc][11]] [[sha1][12]] [[md5][13]]
 
 Upgrade notes: [Click][17]
 
@@ -161,7 +161,7 @@ is mentioned above.
 3.. Verify package with `.asc` signature file
 
 {% highlight bash %}
-gpg --verify apache-libcloud-2.0.0rc2.tar.bz2.asc
+gpg --verify apache-libcloud-2.1.0.tar.bz2.asc
 gpg: Signature made Wed Oct  6 15:31:35 2010 CDT using DSA key ID 42721F00
 gpg: Good signature from "Paul Querna <...>"
 gpg:                 aka "Paul Querna <...>"
@@ -174,31 +174,31 @@ Primary key fingerprint: 39F6 691A 0ECF
 Example of a bad signature:
 
 {% highlight bash %}
-gpg --verify apache-libcloud-2.0.0rc2.tar.bz2.asc
+gpg --verify apache-libcloud-2.1.0.tar.bz2.asc
 gpg: Signature made Wed Oct  6 15:31:35 2010 CDT using DSA key ID 42721F00
 gpg: BAD signature from "Paul Querna <...>"
 {% endhighlight %}
 
 [1]: {{ page.url }}
-[2]: http://www.apache.org/dyn/closer.cgi/libcloud/apache-libcloud-2.0.0rc2.tar.gz
-[3]: http://www.apache.org/dist/libcloud/apache-libcloud-2.0.0rc2.tar.gz.asc
-[4]: http://www.apache.org/dist/libcloud/apache-libcloud-2.0.0rc2.tar.gz.sha1
-[5]: http://www.apache.org/dist/libcloud/apache-libcloud-2.0.0rc2.tar.gz.md5
-[6]: http://www.apache.org/dyn/closer.cgi/libcloud/apache-libcloud-2.0.0rc2.tar.bz2
-[7]: http://www.apache.org/dist/libcloud/apache-libcloud-2.0.0rc2.tar.bz2.asc
-[8]: http://www.apache.org/dist/libcloud/apache-libcloud-2.0.0rc2.tar.bz2.sha1
-[9]: http://www.apache.org/dist/libcloud/apache-libcloud-2.0.0rc2.tar.bz2.md5
-[10]: http://www.apache.org/dyn/closer.cgi/libcloud/apache-libcloud-2.0.0rc2.zip
-[11]: http://www.apache.org/dist/libcloud/apache-libcloud-2.0.0rc2.zip.asc
-[12]: http://www.apache.org/dist/libcloud/apache-libcloud-2.0.0rc2.zip.sha1
-[13]: http://www.apache.org/dist/libcloud/apache-libcloud-2.0.0rc2.zip.md5
+[2]: http://www.apache.org/dyn/closer.cgi/libcloud/apache-libcloud-2.1.0.tar.gz
+[3]: http://www.apache.org/dist/libcloud/apache-libcloud-2.1.0.tar.gz.asc
+[4]: http://www.apache.org/dist/libcloud/apache-libcloud-2.1.0.tar.gz.sha1
+[5]: http://www.apache.org/dist/libcloud/apache-libcloud-2.1.0.tar.gz.md5
+[6]: http://www.apache.org/dyn/closer.cgi/libcloud/apache-libcloud-2.1.0.tar.bz2
+[7]: http://www.apache.org/dist/libcloud/apache-libcloud-2.1.0.tar.bz2.asc
+[8]: http://www.apache.org/dist/libcloud/apache-libcloud-2.1.0.tar.bz2.sha1
+[9]: http://www.apache.org/dist/libcloud/apache-libcloud-2.1.0.tar.bz2.md5
+[10]: http://www.apache.org/dyn/closer.cgi/libcloud/apache-libcloud-2.1.0.zip
+[11]: http://www.apache.org/dist/libcloud/apache-libcloud-2.1.0.zip.asc
+[12]: http://www.apache.org/dist/libcloud/apache-libcloud-2.1.0.zip.sha1
+[13]: http://www.apache.org/dist/libcloud/apache-libcloud-2.1.0.zip.md5
 [14]: https://www.apache.org/dist/libcloud/KEYS
 [15]: http://archive.apache.org/dist/libcloud/
 [16]: https://libcloud.readthedocs.org/en/latest/upgrade_notes.html
-[17]: https://libcloud.readthedocs.org/en/latest/upgrade_notes.html#libcloud-2.0.0rc2
+[17]: https://libcloud.readthedocs.org/en/latest/upgrade_notes.html#libcloud-2.1.0
 [18]: https://github.com/apache/libcloud
 [19]: https://git-wip-us.apache.org/repos/asf/libcloud.git
-[20]: http://www.apache.org/dyn/closer.cgi/libcloud/apache-libcloud-2.0.0rc2-rc2.tar.gz
-[21]: http://www.apache.org/dist/libcloud/apache-libcloud-2.0.0rc2-rc2.tar.gz.asc
-[22]: http://www.apache.org/dist/libcloud/apache-libcloud-2.0.0rc2-rc2.tar.gz.sha1
-[23]: http://www.apache.org/dist/libcloud/apache-libcloud-2.0.0rc2-rc2.tar.gz.md5
+[20]: http://www.apache.org/dyn/closer.cgi/libcloud/apache-libcloud-2.1.0.tar.gz
+[21]: http://www.apache.org/dist/libcloud/apache-libcloud-2.1.0.tar.gz.asc
+[22]: http://www.apache.org/dist/libcloud/apache-libcloud-2.1.0.tar.gz.sha1
+[23]: http://www.apache.org/dist/libcloud/apache-libcloud-2.1.0.tar.gz.md5

Modified: libcloud/site/trunk/source/index.html
URL: http://svn.apache.org/viewvc/libcloud/site/trunk/source/index.html?rev=1802206&r1=1802205&r2=1802206&view=diff
==============================================================================
--- libcloud/site/trunk/source/index.html (original)
+++ libcloud/site/trunk/source/index.html Mon Jul 17 20:03:07 2017
@@ -81,7 +81,7 @@ javascript_files:
     <div class="row section row-2">
       <div class="col-md-6">
         <h3>Installation</h3>
-        <p>Latest stable version: <a href="https://pypi.python.org/pypi/apache-libcloud/2.0.0rc2"
target="_blank">2.0.0rc2</a></p>
+        <p>Latest stable version: <a href="https://pypi.python.org/pypi/apache-libcloud/2.1.0"
target="_blank">2.1.0</a></p>
         <p><code>pip install apache-libcloud</code></p>
         <p>Or <a href="downloads.html">download it from our servers</a>
and install it manually.</p>
       </div>



Mime
View raw message