jclouds-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Nikola Knezevic <laladelausa...@gmail.com>
Subject Problem with S3: SEVERE: error after writing 946176/16777216 bytes
Date Tue, 14 Apr 2015 16:26:31 GMT
Hi,

we're having a problem with writing to S3, where quite frequently we
get the above exception, reproduced here in its entirety:

SEVERE: error after writing 4534272/16777216 bytes to
http://kne.recreate.container.s3.amazonaws.com/testblob-16777216byte-tid0-it20
java.io.IOException: Error writing request body to server
    at sun.net.www.protocol.http.HttpURLConnection$StreamingOutputStream.checkError(HttpURLConnection.java:3205)
    at sun.net.www.protocol.http.HttpURLConnection$StreamingOutputStream.write(HttpURLConnection.java:3188)
    at com.google.common.io.CountingOutputStream.write(CountingOutputStream.java:53)
    at com.google.common.io.ByteStreams.copy(ByteStreams.java:179)
    at org.jclouds.http.internal.JavaUrlHttpCommandExecutorService.writePayloadToConnection(JavaUrlHttpCommandExecutorService.java:297)
    at org.jclouds.http.internal.JavaUrlHttpCommandExecutorService.convert(JavaUrlHttpCommandExecutorService.java:170)
    at org.jclouds.http.internal.JavaUrlHttpCommandExecutorService.convert(JavaUrlHttpCommandExecutorService.java:64)
    at org.jclouds.http.internal.BaseHttpCommandExecutorService.invoke(BaseHttpCommandExecutorService.java:91)
    at org.jclouds.rest.internal.InvokeHttpMethod.invoke(InvokeHttpMethod.java:90)
    at org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:73)
    at org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:44)
    at org.jclouds.rest.internal.DelegatesToInvocationFunction.handle(DelegatesToInvocationFunction.java:156)
    at org.jclouds.rest.internal.DelegatesToInvocationFunction.invoke(DelegatesToInvocationFunction.java:123)
    at com.sun.proxy.$Proxy52.putObject(Unknown Source)
    at org.jclouds.s3.blobstore.S3BlobStore.putBlob(S3BlobStore.java:273)
    at org.jclouds.aws.s3.blobstore.AWSS3BlobStore.putBlob(AWSS3BlobStore.java:94)
    at org.jclouds.s3.blobstore.S3BlobStore.putBlob(S3BlobStore.java:246)
    at net.soba143.knl.Recreate$1.run(Recreate.java:202)
    at java.lang.Thread.run(Thread.java:745)

Apr 14, 2015 4:14:10 PM org.jclouds.logging.jdk.JDKLogger logError
SEVERE: Cannot retry after server error, command is not replayable:
[method=org.jclouds.aws.s3.AWSS3Client.public abstract
java.lang.String
org.jclouds.s3.S3Client.putObject(java.lang.String,org.jclouds.s3.domain.S3Object,org.jclouds.s3.options.PutObjectOptions[])[kne.recreate.container,
[metadata=[key=testblob-16777216byte-tid0-it20, bucket=null, uri=null,
eTag=null, cacheControl=null,
contentMetadata=[contentDisposition=null, contentEncoding=null,
contentLanguage=null, contentLength=16777216, contentMD5=null,
contentType=application/unknown, expires=null], lastModified=null,
owner=null, storageClass=STANDARD, userMetadata={}]],
[Lorg.jclouds.s3.options.PutObjectOptions;@1c714f0e], request=PUT
http://kne.recreate.container.s3.amazonaws.com/testblob-16777216byte-tid0-it20
HTTP/1.1]
[WARNING]
org.jclouds.http.HttpResponseException: Error writing request body to
server connecting to PUT
http://kne.recreate.container.s3.amazonaws.com/testblob-16777216byte-tid0-it20
HTTP/1.1
    at org.jclouds.http.internal.BaseHttpCommandExecutorService.invoke(BaseHttpCommandExecutorService.java:113)
    at org.jclouds.rest.internal.InvokeHttpMethod.invoke(InvokeHttpMethod.java:90)
    at org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:73)
    at org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:44)
    at org.jclouds.rest.internal.DelegatesToInvocationFunction.handle(DelegatesToInvocationFunction.java:156)
    at org.jclouds.rest.internal.DelegatesToInvocationFunction.invoke(DelegatesToInvocationFunction.java:123)
    at com.sun.proxy.$Proxy52.putObject(Unknown Source)
    at org.jclouds.s3.blobstore.S3BlobStore.putBlob(S3BlobStore.java:273)
    at org.jclouds.aws.s3.blobstore.AWSS3BlobStore.putBlob(AWSS3BlobStore.java:94)
    at org.jclouds.s3.blobstore.S3BlobStore.putBlob(S3BlobStore.java:246)
    at com.ibm.icstore.Recreate$1.run(Recreate.java:202)
    at java.lang.Thread.run(Thread.java:745)
Caused by: java.io.IOException: Error writing request body to server
    at sun.net.www.protocol.http.HttpURLConnection$StreamingOutputStream.checkError(HttpURLConnection.java:3205)
    at sun.net.www.protocol.http.HttpURLConnection$StreamingOutputStream.write(HttpURLConnection.java:3188)
    at com.google.common.io.CountingOutputStream.write(CountingOutputStream.java:53)
    at com.google.common.io.ByteStreams.copy(ByteStreams.java:179)
    at org.jclouds.http.internal.JavaUrlHttpCommandExecutorService.writePayloadToConnection(JavaUrlHttpCommandExecutorService.java:297)
    at org.jclouds.http.internal.JavaUrlHttpCommandExecutorService.convert(JavaUrlHttpCommandExecutorService.java:170)
    at org.jclouds.http.internal.JavaUrlHttpCommandExecutorService.convert(JavaUrlHttpCommandExecutorService.java:64)
    at org.jclouds.http.internal.BaseHttpCommandExecutorService.invoke(BaseHttpCommandExecutorService.java:91)
    ... 11 more



The test spawns multiple threads and then each thread starts writing
16MB blobs to S3. Each blob is differently named, and the crux of the
code that does the writing is below. Note that this doesn't fail every
time, but it fails once in 2-3 runs (it fails more often on
Linux+OpenJDK than on Mac+Oracle JDK). Verified on both jclouds 1.8
and 1.9. Also, I manually delete the container before the test.

final int mytid = tid;
Random random = new Random(Thread.currentThread().hashCode());

for (int i = 1; i <= invocationCounter; i++)
{
    System.out.println("Starting round " + i);

    final BlobStore blobStore1 = blobStore;

    final String blobName = "testblob-" + blobSize + "byte" + "-tid" +
mytid + "-it" + i;
    final Blob blob = blobStore1.blobBuilder(blobName)
                        .payload(new FakeInputStream(random.nextInt(),
blobSize))
                        .contentLength(blobSize)
                        .build();

    assertNotNull(blob);

    final String etag = blobStore.putBlob(containerName, blob);
    System.out.println(Thread.currentThread().getId() + " putBlob
returns " + etag);
}

(the rest of the code is here: https://gist.github.com/knl/3b708bca6b31df629c38)

>From our initial investigation, S3 sends us a TCP RST after exchanging
many ACKs and after we send a bunch of 1-20k requests. I tried
fiddling with soTimeout and others, but to no avail.

Has anyone seen a similar problem? If not, what should we do to
prevent this from happening? All our runs on the CI platform fail
because a longer variant of this test keeps failing :(

Thanks,
Nikola

Mime
View raw message