libcloud-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chris Richards <>
Subject How to shorten file download time?
Date Tue, 02 Sep 2014 15:30:40 GMT
Howdy. I've noticed a variance in the download time of a file depending on
the method of download, and I'm hoping to shave off overhead. I'm using the
standard s3 provider. That stats I present are consistent between my office
and my home within +/-100 ms.  In shortened form:

Timing via driver.get_container().get_object().download()
get_container: 431.4596652984619 ms
get_object: 808.0205917358398 ms
download: 8257.043838500977 ms
Complete, downloaded 8.15 MB

Timing via driver.get_object().download()
get_object: 811.8221759796143 ms
download: 4801.661729812622 ms
Complete, downloaded 8.15 MB

In the first case, it appears that getting the container has significant
overhead and should be avoided if possible--which I can do--and trims
400-500 ms per download (for small files, this is significant). Is my
observation and conclusion correct?

In the second case, I want to examine is the .get_object() requirement to
download a file. This adds another significant overhead on the order of
700-900 ms. Is there a way to bypass this?  I have many small files where
the .get_object() time exceeds that of the .download() time!

import std.newbie.disclaimer


  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message