jclouds-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Andrew Gaul <g...@apache.org>
Subject Re: Thread being killed
Date Fri, 05 Dec 2014 17:12:06 GMT
[Moving from the dev to the user mailing list.]

The SEVERE log message is not as alarming as you might suspect; JDK
logger maps error level logs to severe level.  It appears that you are
using the deprecated async jclouds interface instead of the sync
interface; are you consuming the error response via Future.get?
Otherwise, can you minimize a test case so we can diagnose this further?

On Fri, Dec 05, 2014 at 10:41:48AM -0600, Peter Halliday wrote:
> I’m using JClouds BlobStore in an application that’s using it to send files to S3.
 It does this in batches like after a snapshot is run on a database.  And then all of the
files are transferred to S3.  We were testing error states.  And we were testing changing
a User’s Access credentials from inactive to active and back.  When we go from active to
inactive credentials during this transfer to S3.  Our thread gets killed.  The last log entry
is from JClouds.  Our code is literally catching all errors, and so nothing is being thrown
at that time.  Our application is in clojure.  You can see in the log, that we upload successfully
fine, and then after we switch to invalid we “correct” get errors that get returned to
our application.  However, then we hit the SEVERE log line, and then our thread is dead and
never comes back :-( 
> 
> 
> 
> DEBUG [clojure-agent-send-off-pool-9] 2014-12-05 16:29:14,398 Sending message '["172.31.10.206",{"destination_id":"380a2db2ac2042c5bd43043d8ae53b6f","type":"sstable","state":"running","details":{"filename":"OpsCenter-rollups300-jb-96-Data.db","keyspace":"OpsCenter","cf":"rollups300"},"parent_req":"eb335075-765a-46b0-bcb8-89188d93cb1d"}]'
to channel '/snapshot/status'.
> DEBUG [async-dispatch-31] 2014-12-05 16:29:14,496 Uploading file: snapshots/728139a7-093b-40f7-96a9-767a178fa5a5/sstables/OpsCenter-rollups300-jb-98-CompressionInfo.db
> DEBUG [async-dispatch-31] 2014-12-05 16:29:14,498 Throttle value requested is out of
range, using default.
> DEBUG [clojure-agent-send-off-pool-9] 2014-12-05 16:29:14,695 Sending message '["172.31.10.206",{"destination_id":"380a2db2ac2042c5bd43043d8ae53b6f","type":"sstable","state":"running","details":{"filename":"OpsCenter-rollups300-jb-98-CompressionInfo.db","keyspace":"OpsCenter","cf":"rollups300"},"parent_req":"eb335075-765a-46b0-bcb8-89188d93cb1d"}]'
to channel '/snapshot/status'.
> DEBUG [async-dispatch-31] 2014-12-05 16:29:14,797 Uploading file: snapshots/728139a7-093b-40f7-96a9-767a178fa5a5/sstables/OpsCenter-rollups300-jb-98-Index.db
> DEBUG [async-dispatch-31] 2014-12-05 16:29:14,809 Throttle value requested is out of
range, using default.
> DEBUG [clojure-agent-send-off-pool-9] 2014-12-05 16:29:15,105 Sending message '["172.31.10.206",{"destination_id":"380a2db2ac2042c5bd43043d8ae53b6f","type":"sstable","state":"running","details":{"filename":"OpsCenter-rollups300-jb-98-Index.db","keyspace":"OpsCenter","cf":"rollups300"},"parent_req":"eb335075-765a-46b0-bcb8-89188d93cb1d"}]'
to channel '/snapshot/status'.
> DEBUG [clojure-agent-send-off-pool-9] 2014-12-05 16:29:15,204 Sending message '["172.31.10.206",{"destination_id":"380a2db2ac2042c5bd43043d8ae53b6f","type":"sstable","state":"running","details":{"filename":"OpsCenter-rollups300-jb-97-Statistics.db","keyspace":"OpsCenter","cf":"rollups300"},"parent_req":"eb335075-765a-46b0-bcb8-89188d93cb1d"}]'
to channel '/snapshot/status'.
> DEBUG [clojure-agent-send-off-pool-9] 2014-12-05 16:29:15,305 Sending message '["172.31.10.206",{"destination_id":"380a2db2ac2042c5bd43043d8ae53b6f","type":"sstable","state":"running","details":{"filename":"OpsCenter-rollups300-jb-96-Index.db","keyspace":"OpsCenter","cf":"rollups300"},"parent_req":"eb335075-765a-46b0-bcb8-89188d93cb1d"}]'
to channel '/snapshot/status'.
> DEBUG [clojure-agent-send-off-pool-9] 2014-12-05 16:29:15,406 Sending message '["172.31.10.206",{"destination_id":"380a2db2ac2042c5bd43043d8ae53b6f","type":"sstable","state":"running","details":{"filename":"OpsCenter-rollups300-jb-97-Summary.db","keyspace":"OpsCenter","cf":"rollups300"},"parent_req":"eb335075-765a-46b0-bcb8-89188d93cb1d"}]'
to channel '/snapshot/status'.
> ERROR [async-dispatch-31] 2014-12-05 16:29:15,503 Got exception HEAD https://phalliday-opscenter-backups-4.s3.amazonaws.com/snapshots/728139a7-093b-40f7-96a9-767a178fa5a5/sstables/OpsCenter-rollups300-jb-96-Summary.db
HTTP/1.1 -> HTTP/1.1 403 Forbidden while sending opsagent.backups.entities.SSTable@48fee98c
to :380a2db2ac2042c5bd43043d8ae53b6f
> org.jclouds.rest.AuthorizationException: HEAD https://phalliday-opscenter-backups-4.s3.amazonaws.com/snapshots/728139a7-093b-40f7-96a9-767a178fa5a5/sstables/OpsCenter-rollups300-jb-96-Summary.db
HTTP/1.1 -> HTTP/1.1 403 Forbidden
> 	at org.jclouds.aws.handlers.ParseAWSErrorFromXmlContent.refineException(ParseAWSErrorFromXmlContent.java:122)
> 	at org.jclouds.s3.handlers.ParseS3ErrorFromXmlContent.refineException(ParseS3ErrorFromXmlContent.java:94)
> 	at org.jclouds.aws.handlers.ParseAWSErrorFromXmlContent.handleError(ParseAWSErrorFromXmlContent.java:89)
> 	at org.jclouds.http.handlers.DelegatingErrorHandler.handleError(DelegatingErrorHandler.java:65)
> 	at org.jclouds.http.internal.BaseHttpCommandExecutorService.shouldContinue(BaseHttpCommandExecutorService.java:135)
> 	at org.jclouds.http.internal.BaseHttpCommandExecutorService.invoke(BaseHttpCommandExecutorService.java:105)
> 	at org.jclouds.rest.internal.InvokeSyncToAsyncHttpMethod.invoke(InvokeSyncToAsyncHttpMethod.java:128)
> 	at org.jclouds.rest.internal.InvokeSyncToAsyncHttpMethod.apply(InvokeSyncToAsyncHttpMethod.java:94)
> 	at org.jclouds.rest.internal.InvokeSyncToAsyncHttpMethod.apply(InvokeSyncToAsyncHttpMethod.java:55)
> 	at org.jclouds.rest.internal.DelegatesToInvocationFunction.handle(DelegatesToInvocationFunction.java:156)
> 	at org.jclouds.rest.internal.DelegatesToInvocationFunction.invoke(DelegatesToInvocationFunction.java:123)
> 	at com.sun.proxy.$Proxy48.objectExists(Unknown Source)
> 	at org.jclouds.s3.blobstore.S3BlobStore.blobExists(S3BlobStore.java:173)
> 	at org.jclouds.blobstore2$blob_exists_QMARK_.invoke(blobstore2.clj:238)
> 	at opsagent.backups.destinations$create_blob.invoke(destinations.clj:43)
> 	at opsagent.backups.destinations$eval15431$fn__15460.invoke(destinations.clj:123)
> 	at opsagent.backups.destinations$eval15244$fn__15245$G__15235__15254.invoke(destinations.clj:19)
> 	at opsagent.backups.staging$start_staging_BANG_$fn__15649$state_machine__13248__auto____15650$fn__15655$fn__15683.invoke(staging.clj:46)
> 	at opsagent.backups.staging$start_staging_BANG_$fn__15649$state_machine__13248__auto____15650$fn__15655.invoke(staging.clj:44)
> 	at opsagent.backups.staging$start_staging_BANG_$fn__15649$state_machine__13248__auto____15650.invoke(staging.clj:40)
> 	at clojure.core.async.impl.ioc_macros$run_state_machine.invoke(ioc_macros.clj:940)
> 	at clojure.core.async.impl.ioc_macros$run_state_machine_wrapped.invoke(ioc_macros.clj:944)
> 	at clojure.core.async.impl.ioc_macros$take_BANG_$fn__13264.invoke(ioc_macros.clj:953)
> 	at clojure.core.async.impl.channels.ManyToManyChannel$fn__8952.invoke(channels.clj:133)
> 	at clojure.lang.AFn.run(AFn.java:24)
> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> 	at java.lang.Thread.run(Thread.java:744)
> Caused by: org.jclouds.http.HttpResponseException: request: HEAD https://phalliday-opscenter-backups-4.s3.amazonaws.com/snapshots/728139a7-093b-40f7-96a9-767a178fa5a5/sstables/OpsCenter-rollups300-jb-96-Summary.db
HTTP/1.1 failed with response: HTTP/1.1 403 Forbidden
> 	at org.jclouds.aws.handlers.ParseAWSErrorFromXmlContent.handleError(ParseAWSErrorFromXmlContent.java:63)
> 	... 25 more
> DEBUG [clojure-agent-send-off-pool-9] 2014-12-05 16:29:15,504 Sending message '["172.31.10.206",{"details":{"error":"HEAD
https:\/\/phalliday-opscenter-backups-4.s3.amazonaws.com\/snapshots\/728139a7-093b-40f7-96a9-767a178fa5a5\/sstables\/OpsCenter-rollups300-jb-96-Summary.db
HTTP\/1.1 -> HTTP\/1.1 403 Forbidden","filename":"OpsCenter-rollups300-jb-96-Summary.db","keyspace":"OpsCenter","cf":"rollups300"},"state":"running","type":"sstable","destination_id":"380a2db2ac2042c5bd43043d8ae53b6f","parent_req":"eb335075-765a-46b0-bcb8-89188d93cb1d"}]'
to channel '/snapshot/status'.
> ERROR [async-dispatch-31] 2014-12-05 16:29:15,600 Got exception HEAD https://phalliday-opscenter-backups-4.s3.amazonaws.com/snapshots/728139a7-093b-40f7-96a9-767a178fa5a5/sstables/OpsCenter-rollups300-jb-96-Filter.db
HTTP/1.1 -> HTTP/1.1 403 Forbidden while sending opsagent.backups.entities.SSTable@91afeb42
to :380a2db2ac2042c5bd43043d8ae53b6f
> org.jclouds.rest.AuthorizationException: HEAD https://phalliday-opscenter-backups-4.s3.amazonaws.com/snapshots/728139a7-093b-40f7-96a9-767a178fa5a5/sstables/OpsCenter-rollups300-jb-96-Filter.db
HTTP/1.1 -> HTTP/1.1 403 Forbidden
> 	at org.jclouds.aws.handlers.ParseAWSErrorFromXmlContent.refineException(ParseAWSErrorFromXmlContent.java:122)
> 	at org.jclouds.s3.handlers.ParseS3ErrorFromXmlContent.refineException(ParseS3ErrorFromXmlContent.java:94)
> 	at org.jclouds.aws.handlers.ParseAWSErrorFromXmlContent.handleError(ParseAWSErrorFromXmlContent.java:89)
> 	at org.jclouds.http.handlers.DelegatingErrorHandler.handleError(DelegatingErrorHandler.java:65)
> 	at org.jclouds.http.internal.BaseHttpCommandExecutorService.shouldContinue(BaseHttpCommandExecutorService.java:135)
> 	at org.jclouds.http.internal.BaseHttpCommandExecutorService.invoke(BaseHttpCommandExecutorService.java:105)
> 	at org.jclouds.rest.internal.InvokeSyncToAsyncHttpMethod.invoke(InvokeSyncToAsyncHttpMethod.java:128)
> 	at org.jclouds.rest.internal.InvokeSyncToAsyncHttpMethod.apply(InvokeSyncToAsyncHttpMethod.java:94)
> 	at org.jclouds.rest.internal.InvokeSyncToAsyncHttpMethod.apply(InvokeSyncToAsyncHttpMethod.java:55)
> 	at org.jclouds.rest.internal.DelegatesToInvocationFunction.handle(DelegatesToInvocationFunction.java:156)
> 	at org.jclouds.rest.internal.DelegatesToInvocationFunction.invoke(DelegatesToInvocationFunction.java:123)
> 	at com.sun.proxy.$Proxy48.objectExists(Unknown Source)
> 	at org.jclouds.s3.blobstore.S3BlobStore.blobExists(S3BlobStore.java:173)
> 	at org.jclouds.blobstore2$blob_exists_QMARK_.invoke(blobstore2.clj:238)
> 	at opsagent.backups.destinations$create_blob.invoke(destinations.clj:43)
> 	at opsagent.backups.destinations$eval15431$fn__15460.invoke(destinations.clj:123)
> 	at opsagent.backups.destinations$eval15244$fn__15245$G__15235__15254.invoke(destinations.clj:19)
> 	at opsagent.backups.staging$start_staging_BANG_$fn__15649$state_machine__13248__auto____15650$fn__15655$fn__15683.invoke(staging.clj:46)
> 	at opsagent.backups.staging$start_staging_BANG_$fn__15649$state_machine__13248__auto____15650$fn__15655.invoke(staging.clj:44)
> 	at opsagent.backups.staging$start_staging_BANG_$fn__15649$state_machine__13248__auto____15650.invoke(staging.clj:40)
> 	at clojure.core.async.impl.ioc_macros$run_state_machine.invoke(ioc_macros.clj:940)
> 	at clojure.core.async.impl.ioc_macros$run_state_machine_wrapped.invoke(ioc_macros.clj:944)
> 	at clojure.core.async.impl.ioc_macros$take_BANG_$fn__13264.invoke(ioc_macros.clj:953)
> 	at clojure.core.async.impl.channels.ManyToManyChannel$fn__8952.invoke(channels.clj:133)
> 	at clojure.lang.AFn.run(AFn.java:24)
> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> 	at java.lang.Thread.run(Thread.java:744)
> Caused by: org.jclouds.http.HttpResponseException: request: HEAD https://phalliday-opscenter-backups-4.s3.amazonaws.com/snapshots/728139a7-093b-40f7-96a9-767a178fa5a5/sstables/OpsCenter-rollups300-jb-96-Filter.db
HTTP/1.1 failed with response: HTTP/1.1 403 Forbidden
> 	at org.jclouds.aws.handlers.ParseAWSErrorFromXmlContent.handleError(ParseAWSErrorFromXmlContent.java:63)
> 	... 25 more
> DEBUG [clojure-agent-send-off-pool-9] 2014-12-05 16:29:15,601 Sending message '["172.31.10.206",{"details":{"error":"HEAD
https:\/\/phalliday-opscenter-backups-4.s3.amazonaws.com\/snapshots\/728139a7-093b-40f7-96a9-767a178fa5a5\/sstables\/OpsCenter-rollups300-jb-96-Filter.db
HTTP\/1.1 -> HTTP\/1.1 403 Forbidden","filename":"OpsCenter-rollups300-jb-96-Filter.db","keyspace":"OpsCenter","cf":"rollups300"},"state":"running","type":"sstable","destination_id":"380a2db2ac2042c5bd43043d8ae53b6f","parent_req":"eb335075-765a-46b0-bcb8-89188d93cb1d"}]'
to channel '/snapshot/status'.
> ERROR [async-dispatch-31] 2014-12-05 16:29:15,696 Got exception HEAD https://phalliday-opscenter-backups-4.s3.amazonaws.com/snapshots/728139a7-093b-40f7-96a9-767a178fa5a5/sstables/OpsCenter-pdps-jb-114-CompressionInfo.db
HTTP/1.1 -> HTTP/1.1 403 Forbidden while sending opsagent.backups.entities.SSTable@a4c93
to :380a2db2ac2042c5bd43043d8ae53b6f
> org.jclouds.rest.AuthorizationException: HEAD https://phalliday-opscenter-backups-4.s3.amazonaws.com/snapshots/728139a7-093b-40f7-96a9-767a178fa5a5/sstables/OpsCenter-pdps-jb-114-CompressionInfo.db
HTTP/1.1 -> HTTP/1.1 403 Forbidden
> 	at org.jclouds.aws.handlers.ParseAWSErrorFromXmlContent.refineException(ParseAWSErrorFromXmlContent.java:122)
> 	at org.jclouds.s3.handlers.ParseS3ErrorFromXmlContent.refineException(ParseS3ErrorFromXmlContent.java:94)
> 	at org.jclouds.aws.handlers.ParseAWSErrorFromXmlContent.handleError(ParseAWSErrorFromXmlContent.java:89)
> 	at org.jclouds.http.handlers.DelegatingErrorHandler.handleError(DelegatingErrorHandler.java:65)
> 	at org.jclouds.http.internal.BaseHttpCommandExecutorService.shouldContinue(BaseHttpCommandExecutorService.java:135)
> 	at org.jclouds.http.internal.BaseHttpCommandExecutorService.invoke(BaseHttpCommandExecutorService.java:105)
> 	at org.jclouds.rest.internal.InvokeSyncToAsyncHttpMethod.invoke(InvokeSyncToAsyncHttpMethod.java:128)
> 	at org.jclouds.rest.internal.InvokeSyncToAsyncHttpMethod.apply(InvokeSyncToAsyncHttpMethod.java:94)
> 	at org.jclouds.rest.internal.InvokeSyncToAsyncHttpMethod.apply(InvokeSyncToAsyncHttpMethod.java:55)
> 	at org.jclouds.rest.internal.DelegatesToInvocationFunction.handle(DelegatesToInvocationFunction.java:156)
> 	at org.jclouds.rest.internal.DelegatesToInvocationFunction.invoke(DelegatesToInvocationFunction.java:123)
> 	at com.sun.proxy.$Proxy48.objectExists(Unknown Source)
> 	at org.jclouds.s3.blobstore.S3BlobStore.blobExists(S3BlobStore.java:173)
> 	at org.jclouds.blobstore2$blob_exists_QMARK_.invoke(blobstore2.clj:238)
> 	at opsagent.backups.destinations$create_blob.invoke(destinations.clj:43)
> 	at opsagent.backups.destinations$eval15431$fn__15460.invoke(destinations.clj:123)
> 	at opsagent.backups.destinations$eval15244$fn__15245$G__15235__15254.invoke(destinations.clj:19)
> 	at opsagent.backups.staging$start_staging_BANG_$fn__15649$state_machine__13248__auto____15650$fn__15655$fn__15683.invoke(staging.clj:46)
> 	at opsagent.backups.staging$start_staging_BANG_$fn__15649$state_machine__13248__auto____15650$fn__15655.invoke(staging.clj:44)
> 	at opsagent.backups.staging$start_staging_BANG_$fn__15649$state_machine__13248__auto____15650.invoke(staging.clj:40)
> 	at clojure.core.async.impl.ioc_macros$run_state_machine.invoke(ioc_macros.clj:940)
> 	at clojure.core.async.impl.ioc_macros$run_state_machine_wrapped.invoke(ioc_macros.clj:944)
> 	at clojure.core.async.impl.ioc_macros$take_BANG_$fn__13264.invoke(ioc_macros.clj:953)
> 	at clojure.core.async.impl.channels.ManyToManyChannel$fn__8952.invoke(channels.clj:133)
> 	at clojure.lang.AFn.run(AFn.java:24)
> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> 	at java.lang.Thread.run(Thread.java:744)
> Caused by: org.jclouds.http.HttpResponseException: request: HEAD https://phalliday-opscenter-backups-4.s3.amazonaws.com/snapshots/728139a7-093b-40f7-96a9-767a178fa5a5/sstables/OpsCenter-pdps-jb-114-CompressionInfo.db
HTTP/1.1 failed with response: HTTP/1.1 403 Forbidden
> 	at org.jclouds.aws.handlers.ParseAWSErrorFromXmlContent.handleError(ParseAWSErrorFromXmlContent.java:63)
> 	... 25 more
> DEBUG [clojure-agent-send-off-pool-9] 2014-12-05 16:29:15,697 Sending message '["172.31.10.206",{"details":{"error":"HEAD
https:\/\/phalliday-opscenter-backups-4.s3.amazonaws.com\/snapshots\/728139a7-093b-40f7-96a9-767a178fa5a5\/sstables\/OpsCenter-pdps-jb-114-CompressionInfo.db
HTTP\/1.1 -> HTTP\/1.1 403 Forbidden","filename":"OpsCenter-pdps-jb-114-CompressionInfo.db","keyspace":"OpsCenter","cf":"pdps"},"state":"running","type":"sstable","destination_id":"380a2db2ac2042c5bd43043d8ae53b6f","parent_req":"eb335075-765a-46b0-bcb8-89188d93cb1d"}]'
to channel '/snapshot/status'.
> ERROR [async-dispatch-31] 2014-12-05 16:29:15,792 Got exception HEAD https://phalliday-opscenter-backups-4.s3.amazonaws.com/snapshots/728139a7-093b-40f7-96a9-767a178fa5a5/sstables/OpsCenter-pdps-jb-114-Statistics.db
HTTP/1.1 -> HTTP/1.1 403 Forbidden while sending opsagent.backups.entities.SSTable@44e5d97
to :380a2db2ac2042c5bd43043d8ae53b6f
> org.jclouds.rest.AuthorizationException: HEAD https://phalliday-opscenter-backups-4.s3.amazonaws.com/snapshots/728139a7-093b-40f7-96a9-767a178fa5a5/sstables/OpsCenter-pdps-jb-114-Statistics.db
HTTP/1.1 -> HTTP/1.1 403 Forbidden
> 	at org.jclouds.aws.handlers.ParseAWSErrorFromXmlContent.refineException(ParseAWSErrorFromXmlContent.java:122)
> 	at org.jclouds.s3.handlers.ParseS3ErrorFromXmlContent.refineException(ParseS3ErrorFromXmlContent.java:94)
> 	at org.jclouds.aws.handlers.ParseAWSErrorFromXmlContent.handleError(ParseAWSErrorFromXmlContent.java:89)
> 	at org.jclouds.http.handlers.DelegatingErrorHandler.handleError(DelegatingErrorHandler.java:65)
> 	at org.jclouds.http.internal.BaseHttpCommandExecutorService.shouldContinue(BaseHttpCommandExecutorService.java:135)
> 	at org.jclouds.http.internal.BaseHttpCommandExecutorService.invoke(BaseHttpCommandExecutorService.java:105)
> 	at org.jclouds.rest.internal.InvokeSyncToAsyncHttpMethod.invoke(InvokeSyncToAsyncHttpMethod.java:128)
> 	at org.jclouds.rest.internal.InvokeSyncToAsyncHttpMethod.apply(InvokeSyncToAsyncHttpMethod.java:94)
> 	at org.jclouds.rest.internal.InvokeSyncToAsyncHttpMethod.apply(InvokeSyncToAsyncHttpMethod.java:55)
> 	at org.jclouds.rest.internal.DelegatesToInvocationFunction.handle(DelegatesToInvocationFunction.java:156)
> 	at org.jclouds.rest.internal.DelegatesToInvocationFunction.invoke(DelegatesToInvocationFunction.java:123)
> 	at com.sun.proxy.$Proxy48.objectExists(Unknown Source)
> 	at org.jclouds.s3.blobstore.S3BlobStore.blobExists(S3BlobStore.java:173)
> 	at org.jclouds.blobstore2$blob_exists_QMARK_.invoke(blobstore2.clj:238)
> 	at opsagent.backups.destinations$create_blob.invoke(destinations.clj:43)
> 	at opsagent.backups.destinations$eval15431$fn__15460.invoke(destinations.clj:123)
> 	at opsagent.backups.destinations$eval15244$fn__15245$G__15235__15254.invoke(destinations.clj:19)
> 	at opsagent.backups.staging$start_staging_BANG_$fn__15649$state_machine__13248__auto____15650$fn__15655$fn__15683.invoke(staging.clj:46)
> 	at opsagent.backups.staging$start_staging_BANG_$fn__15649$state_machine__13248__auto____15650$fn__15655.invoke(staging.clj:44)
> 	at opsagent.backups.staging$start_staging_BANG_$fn__15649$state_machine__13248__auto____15650.invoke(staging.clj:40)
> 	at clojure.core.async.impl.ioc_macros$run_state_machine.invoke(ioc_macros.clj:940)
> 	at clojure.core.async.impl.ioc_macros$run_state_machine_wrapped.invoke(ioc_macros.clj:944)
> 	at clojure.core.async.impl.ioc_macros$take_BANG_$fn__13264.invoke(ioc_macros.clj:953)
> 	at clojure.core.async.impl.channels.ManyToManyChannel$fn__8952.invoke(channels.clj:133)
> 	at clojure.lang.AFn.run(AFn.java:24)
> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> 	at java.lang.Thread.run(Thread.java:744)
> Caused by: org.jclouds.http.HttpResponseException: request: HEAD https://phalliday-opscenter-backups-4.s3.amazonaws.com/snapshots/728139a7-093b-40f7-96a9-767a178fa5a5/sstables/OpsCenter-pdps-jb-114-Statistics.db
HTTP/1.1 failed with response: HTTP/1.1 403 Forbidden
> 	at org.jclouds.aws.handlers.ParseAWSErrorFromXmlContent.handleError(ParseAWSErrorFromXmlContent.java:63)
> 	... 25 more
> DEBUG [clojure-agent-send-off-pool-9] 2014-12-05 16:29:15,793 Sending message '["172.31.10.206",{"details":{"error":"HEAD
https:\/\/phalliday-opscenter-backups-4.s3.amazonaws.com\/snapshots\/728139a7-093b-40f7-96a9-767a178fa5a5\/sstables\/OpsCenter-pdps-jb-114-Statistics.db
HTTP\/1.1 -> HTTP\/1.1 403 Forbidden","filename":"OpsCenter-pdps-jb-114-Statistics.db","keyspace":"OpsCenter","cf":"pdps"},"state":"running","type":"sstable","destination_id":"380a2db2ac2042c5bd43043d8ae53b6f","parent_req":"eb335075-765a-46b0-bcb8-89188d93cb1d"}]'
to channel '/snapshot/status'.
> ERROR [async-dispatch-31] 2014-12-05 16:29:15,888 Got exception HEAD https://phalliday-opscenter-backups-4.s3.amazonaws.com/snapshots/728139a7-093b-40f7-96a9-767a178fa5a5/sstables/OpsCenter-pdps-jb-114-Data.db
HTTP/1.1 -> HTTP/1.1 403 Forbidden while sending opsagent.backups.entities.SSTable@8ca4473b
to :380a2db2ac2042c5bd43043d8ae53b6f
> org.jclouds.rest.AuthorizationException: HEAD https://phalliday-opscenter-backups-4.s3.amazonaws.com/snapshots/728139a7-093b-40f7-96a9-767a178fa5a5/sstables/OpsCenter-pdps-jb-114-Data.db
HTTP/1.1 -> HTTP/1.1 403 Forbidden
> 	at org.jclouds.aws.handlers.ParseAWSErrorFromXmlContent.refineException(ParseAWSErrorFromXmlContent.java:122)
> 	at org.jclouds.s3.handlers.ParseS3ErrorFromXmlContent.refineException(ParseS3ErrorFromXmlContent.java:94)
> 	at org.jclouds.aws.handlers.ParseAWSErrorFromXmlContent.handleError(ParseAWSErrorFromXmlContent.java:89)
> 	at org.jclouds.http.handlers.DelegatingErrorHandler.handleError(DelegatingErrorHandler.java:65)
> 	at org.jclouds.http.internal.BaseHttpCommandExecutorService.shouldContinue(BaseHttpCommandExecutorService.java:135)
> 	at org.jclouds.http.internal.BaseHttpCommandExecutorService.invoke(BaseHttpCommandExecutorService.java:105)
> 	at org.jclouds.rest.internal.InvokeSyncToAsyncHttpMethod.invoke(InvokeSyncToAsyncHttpMethod.java:128)
> 	at org.jclouds.rest.internal.InvokeSyncToAsyncHttpMethod.apply(InvokeSyncToAsyncHttpMethod.java:94)
> 	at org.jclouds.rest.internal.InvokeSyncToAsyncHttpMethod.apply(InvokeSyncToAsyncHttpMethod.java:55)
> 	at org.jclouds.rest.internal.DelegatesToInvocationFunction.handle(DelegatesToInvocationFunction.java:156)
> 	at org.jclouds.rest.internal.DelegatesToInvocationFunction.invoke(DelegatesToInvocationFunction.java:123)
> 	at com.sun.proxy.$Proxy48.objectExists(Unknown Source)
> 	at org.jclouds.s3.blobstore.S3BlobStore.blobExists(S3BlobStore.java:173)
> 	at org.jclouds.blobstore2$blob_exists_QMARK_.invoke(blobstore2.clj:238)
> 	at opsagent.backups.destinations$create_blob.invoke(destinations.clj:43)
> 	at opsagent.backups.destinations$eval15431$fn__15460.invoke(destinations.clj:123)
> 	at opsagent.backups.destinations$eval15244$fn__15245$G__15235__15254.invoke(destinations.clj:19)
> 	at opsagent.backups.staging$start_staging_BANG_$fn__15649$state_machine__13248__auto____15650$fn__15655$fn__15683.invoke(staging.clj:46)
> 	at opsagent.backups.staging$start_staging_BANG_$fn__15649$state_machine__13248__auto____15650$fn__15655.invoke(staging.clj:44)
> 	at opsagent.backups.staging$start_staging_BANG_$fn__15649$state_machine__13248__auto____15650.invoke(staging.clj:40)
> 	at clojure.core.async.impl.ioc_macros$run_state_machine.invoke(ioc_macros.clj:940)
> 	at clojure.core.async.impl.ioc_macros$run_state_machine_wrapped.invoke(ioc_macros.clj:944)
> 	at clojure.core.async.impl.ioc_macros$take_BANG_$fn__13264.invoke(ioc_macros.clj:953)
> 	at clojure.core.async.impl.channels.ManyToManyChannel$fn__8952.invoke(channels.clj:133)
> 	at clojure.lang.AFn.run(AFn.java:24)
> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> 	at java.lang.Thread.run(Thread.java:744)
> Caused by: org.jclouds.http.HttpResponseException: request: HEAD https://phalliday-opscenter-backups-4.s3.amazonaws.com/snapshots/728139a7-093b-40f7-96a9-767a178fa5a5/sstables/OpsCenter-pdps-jb-114-Data.db
HTTP/1.1 failed with response: HTTP/1.1 403 Forbidden
> 	at org.jclouds.aws.handlers.ParseAWSErrorFromXmlContent.handleError(ParseAWSErrorFromXmlContent.java:63)
> 	... 25 more
> DEBUG [clojure-agent-send-off-pool-9] 2014-12-05 16:29:15,890 Sending message '["172.31.10.206",{"details":{"error":"HEAD
https:\/\/phalliday-opscenter-backups-4.s3.amazonaws.com\/snapshots\/728139a7-093b-40f7-96a9-767a178fa5a5\/sstables\/OpsCenter-pdps-jb-114-Data.db
HTTP\/1.1 -> HTTP\/1.1 403 Forbidden","filename":"OpsCenter-pdps-jb-114-Data.db","keyspace":"OpsCenter","cf":"pdps"},"state":"running","type":"sstable","destination_id":"380a2db2ac2042c5bd43043d8ae53b6f","parent_req":"eb335075-765a-46b0-bcb8-89188d93cb1d"}]'
to channel '/snapshot/status'.
> DEBUG [node-details-1] 2014-12-05 16:29:16,030 Starting real-time collection
> DEBUG [node-details-1] 2014-12-05 16:29:16,030 Collecting real-time data
> DEBUG [clojure-agent-send-off-pool-9] 2014-12-05 16:29:16,034 Sending message '["172.31.10.206",{"os-load":0.07}]'
to channel '/node-details'.
> DEBUG [clojure-agent-send-off-pool-9] 2014-12-05 16:29:16,298 Sending message '["172.31.10.206",{"destination_id":"380a2db2ac2042c5bd43043d8ae53b6f","type":"schema","state":"running","details":{"keyspace":"OpsCenter","filename":"OpsCenter\/schema.json"},"parent_req":"eb335075-765a-46b0-bcb8-89188d93cb1d"}]'
to channel '/snapshot/status'.
> DEBUG [async-dispatch-31] 2014-12-05 16:29:16,402 Uploading file: snapshots/728139a7-093b-40f7-96a9-767a178fa5a5/sstables/OpsCenter-pdps-jb-115-Summary.db
> DEBUG [async-dispatch-31] 2014-12-05 16:29:16,404 Throttle value requested is out of
range, using default.
> ERROR [async-dispatch-31] 2014-12-05 16:29:16,510 Dec 05, 2014 4:29:16 PM org.jclouds.logging.jdk.JDKLogger
logError
> SEVERE: Cannot retry after server error, command is not replayable: [method=org.jclouds.aws.s3.AWSS3AsyncClient.public
abstract com.google.common.util.concurrent.ListenableFuture org.jclouds.s3.S3AsyncClient.putObject(java.lang.String,org.jclouds.s3.domain.S3Object,org.jclouds.s3.options.PutObjectOptions[])[phalliday-opscenter-backups-4,
[metadata=[key=snapshots/728139a7-093b-40f7-96a9-767a178fa5a5/sstables/OpsCenter-pdps-jb-115-Summary.db,
bucket=null, uri=null, eTag=null, cacheControl=null, contentMetadata=[contentDisposition=null,
contentEncoding=null, contentLanguage=null, contentLength=107, contentMD5=null, contentType=application/octet-stream,
expires=null], lastModified=null, owner=null, storageClass=STANDARD, userMetadata={}]], [Lorg.jclouds.s3.options.PutObjectOptions;@4e0ecbfe],
request=PUT https://phalliday-opscenter-backups-4.s3.amazonaws.com/snapshots/728139a7-093b-40f7-96a9-767a178fa5a5/sstables/OpsCenter-pdps-jb-115-Summary.db
HTTP/1.1]
> 
> DEBUG [node-details-1] 2014-12-05 16:29:21,034 Starting real-time collection
> DEBUG [node-details-1] 2014-12-05 16:29:21,035 Collecting real-time data
> DEBUG [node-details-1] 2014-12-05 16:29:21,038 Send requested, but nothing new to send
> 
> 
> 
> 
> 
>  <http://www.datastax.com/>
> PETER HALLIDAY
> OpsCenter | Sr Software Engineer | 607.329.6905 | peter@datastax.com
> 
>  <https://twitter.com/phalliday>  <https://www.linkedin.com/profile/view?id=7030852>

-- 
Andrew Gaul
http://gaul.org/

Mime
View raw message