jclouds-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Andrew Gaul <g...@apache.org>
Subject Re: Problem with S3: SEVERE: error after writing 946176/16777216 bytes
Date Fri, 17 Apr 2015 18:20:38 GMT
On Tue, Apr 14, 2015 at 06:26:31PM +0200, Nikola Knezevic wrote:
> we're having a problem with writing to S3, where quite frequently we
> get the above exception, reproduced here in its entirety:
> ...
> The test spawns multiple threads and then each thread starts writing
> 16MB blobs to S3. Each blob is differently named, and the crux of the
> code that does the writing is below. Note that this doesn't fail every
> time, but it fails once in 2-3 runs (it fails more often on
> Linux+OpenJDK than on Mac+Oracle JDK). Verified on both jclouds 1.8
> and 1.9. Also, I manually delete the container before the test.
> ...
> From our initial investigation, S3 sends us a TCP RST after exchanging
> many ACKs and after we send a bunch of 1-20k requests. I tried
> fiddling with soTimeout and others, but to no avail.
> Has anyone seen a similar problem? If not, what should we do to
> prevent this from happening? All our runs on the CI platform fail
> because a longer variant of this test keeps failing :(

I reproduced your symptoms using an EC2 instance to reduce network
variability.  I found that creating a container with a random name works
around your symptoms.  Container operations have less consistent
semantics than object operations and it is possible that the delete
container from one run conflicts with the object operations from the
subsequent run.  Can you also test this?

You might also have a better experience if you use a reusable Payload
like ByteSource which allows the jclouds retry mechanism to work.
Despite you setting PROPERTY_MAX_RETRIES to 15, the InputStream payload
is not reusable so this is effectively 0.  In the example code you could
use something like TestUtils.randomByteSource[1].

[1] https://github.com/jclouds/jclouds/blob/master/core/src/test/java/org/jclouds/utils/TestUtils.java

Andrew Gaul

View raw message