jclouds-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Peter Halliday <pe...@datastax.com>
Subject Re: Thread being killed
Date Fri, 05 Dec 2014 17:22:19 GMT
My alarm has nothing to do with the word SEVERE, it’s more due to the fact that my thread
dies.  I’m just using the blobstore clojure api, I’m not familiar with the async vs sync
denotation in coordination with that API.


 <http://www.datastax.com/>
PETER HALLIDAY
OpsCenter | Sr Software Engineer | 607.329.6905 | peter@datastax.com <mailto:peter@datastax.com>
 <https://twitter.com/phalliday>  <https://www.linkedin.com/profile/view?id=7030852>
> On Dec 5, 2014, at 11:12 AM, Andrew Gaul <gaul@apache.org <mailto:gaul@apache.org>>
wrote:
> 
> [Moving from the dev to the user mailing list.]
> 
> The SEVERE log message is not as alarming as you might suspect; JDK
> logger maps error level logs to severe level.  It appears that you are
> using the deprecated async jclouds interface instead of the sync
> interface; are you consuming the error response via Future.get?
> Otherwise, can you minimize a test case so we can diagnose this further?
> 
> On Fri, Dec 05, 2014 at 10:41:48AM -0600, Peter Halliday wrote:
>> I’m using JClouds BlobStore in an application that’s using it to send files to
S3.  It does this in batches like after a snapshot is run on a database.  And then all of
the files are transferred to S3.  We were testing error states.  And we were testing changing
a User’s Access credentials from inactive to active and back.  When we go from active to
inactive credentials during this transfer to S3.  Our thread gets killed.  The last log entry
is from JClouds.  Our code is literally catching all errors, and so nothing is being thrown
at that time.  Our application is in clojure.  You can see in the log, that we upload successfully
fine, and then after we switch to invalid we “correct” get errors that get returned to
our application.  However, then we hit the SEVERE log line, and then our thread is dead and
never comes back :-( 
>> 
>> 
>> 
>> DEBUG [clojure-agent-send-off-pool-9] 2014-12-05 16:29:14,398 Sending message '["172.31.10.206",{"destination_id":"380a2db2ac2042c5bd43043d8ae53b6f","type":"sstable","state":"running","details":{"filename":"OpsCenter-rollups300-jb-96-Data.db","keyspace":"OpsCenter","cf":"rollups300"},"parent_req":"eb335075-765a-46b0-bcb8-89188d93cb1d"}]'
to channel '/snapshot/status'.
>> DEBUG [async-dispatch-31] 2014-12-05 16:29:14,496 Uploading file: snapshots/728139a7-093b-40f7-96a9-767a178fa5a5/sstables/OpsCenter-rollups300-jb-98-CompressionInfo.db
>> DEBUG [async-dispatch-31] 2014-12-05 16:29:14,498 Throttle value requested is out
of range, using default.
>> DEBUG [clojure-agent-send-off-pool-9] 2014-12-05 16:29:14,695 Sending message '["172.31.10.206",{"destination_id":"380a2db2ac2042c5bd43043d8ae53b6f","type":"sstable","state":"running","details":{"filename":"OpsCenter-rollups300-jb-98-CompressionInfo.db","keyspace":"OpsCenter","cf":"rollups300"},"parent_req":"eb335075-765a-46b0-bcb8-89188d93cb1d"}]'
to channel '/snapshot/status'.
>> DEBUG [async-dispatch-31] 2014-12-05 16:29:14,797 Uploading file: snapshots/728139a7-093b-40f7-96a9-767a178fa5a5/sstables/OpsCenter-rollups300-jb-98-Index.db
>> DEBUG [async-dispatch-31] 2014-12-05 16:29:14,809 Throttle value requested is out
of range, using default.
>> DEBUG [clojure-agent-send-off-pool-9] 2014-12-05 16:29:15,105 Sending message '["172.31.10.206",{"destination_id":"380a2db2ac2042c5bd43043d8ae53b6f","type":"sstable","state":"running","details":{"filename":"OpsCenter-rollups300-jb-98-Index.db","keyspace":"OpsCenter","cf":"rollups300"},"parent_req":"eb335075-765a-46b0-bcb8-89188d93cb1d"}]'
to channel '/snapshot/status'.
>> DEBUG [clojure-agent-send-off-pool-9] 2014-12-05 16:29:15,204 Sending message '["172.31.10.206",{"destination_id":"380a2db2ac2042c5bd43043d8ae53b6f","type":"sstable","state":"running","details":{"filename":"OpsCenter-rollups300-jb-97-Statistics.db","keyspace":"OpsCenter","cf":"rollups300"},"parent_req":"eb335075-765a-46b0-bcb8-89188d93cb1d"}]'
to channel '/snapshot/status'.
>> DEBUG [clojure-agent-send-off-pool-9] 2014-12-05 16:29:15,305 Sending message '["172.31.10.206",{"destination_id":"380a2db2ac2042c5bd43043d8ae53b6f","type":"sstable","state":"running","details":{"filename":"OpsCenter-rollups300-jb-96-Index.db","keyspace":"OpsCenter","cf":"rollups300"},"parent_req":"eb335075-765a-46b0-bcb8-89188d93cb1d"}]'
to channel '/snapshot/status'.
>> DEBUG [clojure-agent-send-off-pool-9] 2014-12-05 16:29:15,406 Sending message '["172.31.10.206",{"destination_id":"380a2db2ac2042c5bd43043d8ae53b6f","type":"sstable","state":"running","details":{"filename":"OpsCenter-rollups300-jb-97-Summary.db","keyspace":"OpsCenter","cf":"rollups300"},"parent_req":"eb335075-765a-46b0-bcb8-89188d93cb1d"}]'
to channel '/snapshot/status'.
>> ERROR [async-dispatch-31] 2014-12-05 16:29:15,503 Got exception HEAD https://phalliday-opscenter-backups-4.s3.amazonaws.com/snapshots/728139a7-093b-40f7-96a9-767a178fa5a5/sstables/OpsCenter-rollups300-jb-96-Summary.db
<https://phalliday-opscenter-backups-4.s3.amazonaws.com/snapshots/728139a7-093b-40f7-96a9-767a178fa5a5/sstables/OpsCenter-rollups300-jb-96-Summary.db>
HTTP/1.1 -> HTTP/1.1 403 Forbidden while sending opsagent.backups.entities.SSTable@48fee98c
to :380a2db2ac2042c5bd43043d8ae53b6f
>> org.jclouds.rest.AuthorizationException: HEAD https://phalliday-opscenter-backups-4.s3.amazonaws.com/snapshots/728139a7-093b-40f7-96a9-767a178fa5a5/sstables/OpsCenter-rollups300-jb-96-Summary.db
<https://phalliday-opscenter-backups-4.s3.amazonaws.com/snapshots/728139a7-093b-40f7-96a9-767a178fa5a5/sstables/OpsCenter-rollups300-jb-96-Summary.db>
HTTP/1.1 -> HTTP/1.1 403 Forbidden
>> 	at org.jclouds.aws.handlers.ParseAWSErrorFromXmlContent.refineException(ParseAWSErrorFromXmlContent.java:122)
>> 	at org.jclouds.s3.handlers.ParseS3ErrorFromXmlContent.refineException(ParseS3ErrorFromXmlContent.java:94)
>> 	at org.jclouds.aws.handlers.ParseAWSErrorFromXmlContent.handleError(ParseAWSErrorFromXmlContent.java:89)
>> 	at org.jclouds.http.handlers.DelegatingErrorHandler.handleError(DelegatingErrorHandler.java:65)
>> 	at org.jclouds.http.internal.BaseHttpCommandExecutorService.shouldContinue(BaseHttpCommandExecutorService.java:135)
>> 	at org.jclouds.http.internal.BaseHttpCommandExecutorService.invoke(BaseHttpCommandExecutorService.java:105)
>> 	at org.jclouds.rest.internal.InvokeSyncToAsyncHttpMethod.invoke(InvokeSyncToAsyncHttpMethod.java:128)
>> 	at org.jclouds.rest.internal.InvokeSyncToAsyncHttpMethod.apply(InvokeSyncToAsyncHttpMethod.java:94)
>> 	at org.jclouds.rest.internal.InvokeSyncToAsyncHttpMethod.apply(InvokeSyncToAsyncHttpMethod.java:55)
>> 	at org.jclouds.rest.internal.DelegatesToInvocationFunction.handle(DelegatesToInvocationFunction.java:156)
>> 	at org.jclouds.rest.internal.DelegatesToInvocationFunction.invoke(DelegatesToInvocationFunction.java:123)
>> 	at com.sun.proxy.$Proxy48.objectExists(Unknown Source)
>> 	at org.jclouds.s3.blobstore.S3BlobStore.blobExists(S3BlobStore.java:173)
>> 	at org.jclouds.blobstore2$blob_exists_QMARK_.invoke(blobstore2.clj:238)
>> 	at opsagent.backups.destinations$create_blob.invoke(destinations.clj:43)
>> 	at opsagent.backups.destinations$eval15431$fn__15460.invoke(destinations.clj:123)
>> 	at opsagent.backups.destinations$eval15244$fn__15245$G__15235__15254.invoke(destinations.clj:19)
>> 	at opsagent.backups.staging$start_staging_BANG_$fn__15649$state_machine__13248__auto____15650$fn__15655$fn__15683.invoke(staging.clj:46)
>> 	at opsagent.backups.staging$start_staging_BANG_$fn__15649$state_machine__13248__auto____15650$fn__15655.invoke(staging.clj:44)
>> 	at opsagent.backups.staging$start_staging_BANG_$fn__15649$state_machine__13248__auto____15650.invoke(staging.clj:40)
>> 	at clojure.core.async.impl.ioc_macros$run_state_machine.invoke(ioc_macros.clj:940)
>> 	at clojure.core.async.impl.ioc_macros$run_state_machine_wrapped.invoke(ioc_macros.clj:944)
>> 	at clojure.core.async.impl.ioc_macros$take_BANG_$fn__13264.invoke(ioc_macros.clj:953)
>> 	at clojure.core.async.impl.channels.ManyToManyChannel$fn__8952.invoke(channels.clj:133)
>> 	at clojure.lang.AFn.run(AFn.java:24)
>> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>> 	at java.lang.Thread.run(Thread.java:744)
>> Caused by: org.jclouds.http.HttpResponseException: request: HEAD https://phalliday-opscenter-backups-4.s3.amazonaws.com/snapshots/728139a7-093b-40f7-96a9-767a178fa5a5/sstables/OpsCenter-rollups300-jb-96-Summary.db
<https://phalliday-opscenter-backups-4.s3.amazonaws.com/snapshots/728139a7-093b-40f7-96a9-767a178fa5a5/sstables/OpsCenter-rollups300-jb-96-Summary.db>
HTTP/1.1 failed with response: HTTP/1.1 403 Forbidden
>> 	at org.jclouds.aws.handlers.ParseAWSErrorFromXmlContent.handleError(ParseAWSErrorFromXmlContent.java:63)
>> 	... 25 more
>> DEBUG [clojure-agent-send-off-pool-9] 2014-12-05 16:29:15,504 Sending message '["172.31.10.206",{"details":{"error":"HEAD
https:\/\/phalliday-opscenter-backups-4.s3.amazonaws.com <http://phalliday-opscenter-backups-4.s3.amazonaws.com/>\/snapshots\/728139a7-093b-40f7-96a9-767a178fa5a5\/sstables\/OpsCenter-rollups300-jb-96-Summary.db
HTTP\/1.1 -> HTTP\/1.1 403 Forbidden","filename":"OpsCenter-rollups300-jb-96-Summary.db","keyspace":"OpsCenter","cf":"rollups300"},"state":"running","type":"sstable","destination_id":"380a2db2ac2042c5bd43043d8ae53b6f","parent_req":"eb335075-765a-46b0-bcb8-89188d93cb1d"}]'
to channel '/snapshot/status'.
>> ERROR [async-dispatch-31] 2014-12-05 16:29:15,600 Got exception HEAD https://phalliday-opscenter-backups-4.s3.amazonaws.com/snapshots/728139a7-093b-40f7-96a9-767a178fa5a5/sstables/OpsCenter-rollups300-jb-96-Filter.db
<https://phalliday-opscenter-backups-4.s3.amazonaws.com/snapshots/728139a7-093b-40f7-96a9-767a178fa5a5/sstables/OpsCenter-rollups300-jb-96-Filter.db>
HTTP/1.1 -> HTTP/1.1 403 Forbidden while sending opsagent.backups.entities.SSTable@91afeb42
to :380a2db2ac2042c5bd43043d8ae53b6f
>> org.jclouds.rest.AuthorizationException: HEAD https://phalliday-opscenter-backups-4.s3.amazonaws.com/snapshots/728139a7-093b-40f7-96a9-767a178fa5a5/sstables/OpsCenter-rollups300-jb-96-Filter.db
<https://phalliday-opscenter-backups-4.s3.amazonaws.com/snapshots/728139a7-093b-40f7-96a9-767a178fa5a5/sstables/OpsCenter-rollups300-jb-96-Filter.db>
HTTP/1.1 -> HTTP/1.1 403 Forbidden
>> 	at org.jclouds.aws.handlers.ParseAWSErrorFromXmlContent.refineException(ParseAWSErrorFromXmlContent.java:122)
>> 	at org.jclouds.s3.handlers.ParseS3ErrorFromXmlContent.refineException(ParseS3ErrorFromXmlContent.java:94)
>> 	at org.jclouds.aws.handlers.ParseAWSErrorFromXmlContent.handleError(ParseAWSErrorFromXmlContent.java:89)
>> 	at org.jclouds.http.handlers.DelegatingErrorHandler.handleError(DelegatingErrorHandler.java:65)
>> 	at org.jclouds.http.internal.BaseHttpCommandExecutorService.shouldContinue(BaseHttpCommandExecutorService.java:135)
>> 	at org.jclouds.http.internal.BaseHttpCommandExecutorService.invoke(BaseHttpCommandExecutorService.java:105)
>> 	at org.jclouds.rest.internal.InvokeSyncToAsyncHttpMethod.invoke(InvokeSyncToAsyncHttpMethod.java:128)
>> 	at org.jclouds.rest.internal.InvokeSyncToAsyncHttpMethod.apply(InvokeSyncToAsyncHttpMethod.java:94)
>> 	at org.jclouds.rest.internal.InvokeSyncToAsyncHttpMethod.apply(InvokeSyncToAsyncHttpMethod.java:55)
>> 	at org.jclouds.rest.internal.DelegatesToInvocationFunction.handle(DelegatesToInvocationFunction.java:156)
>> 	at org.jclouds.rest.internal.DelegatesToInvocationFunction.invoke(DelegatesToInvocationFunction.java:123)
>> 	at com.sun.proxy.$Proxy48.objectExists(Unknown Source)
>> 	at org.jclouds.s3.blobstore.S3BlobStore.blobExists(S3BlobStore.java:173)
>> 	at org.jclouds.blobstore2$blob_exists_QMARK_.invoke(blobstore2.clj:238)
>> 	at opsagent.backups.destinations$create_blob.invoke(destinations.clj:43)
>> 	at opsagent.backups.destinations$eval15431$fn__15460.invoke(destinations.clj:123)
>> 	at opsagent.backups.destinations$eval15244$fn__15245$G__15235__15254.invoke(destinations.clj:19)
>> 	at opsagent.backups.staging$start_staging_BANG_$fn__15649$state_machine__13248__auto____15650$fn__15655$fn__15683.invoke(staging.clj:46)
>> 	at opsagent.backups.staging$start_staging_BANG_$fn__15649$state_machine__13248__auto____15650$fn__15655.invoke(staging.clj:44)
>> 	at opsagent.backups.staging$start_staging_BANG_$fn__15649$state_machine__13248__auto____15650.invoke(staging.clj:40)
>> 	at clojure.core.async.impl.ioc_macros$run_state_machine.invoke(ioc_macros.clj:940)
>> 	at clojure.core.async.impl.ioc_macros$run_state_machine_wrapped.invoke(ioc_macros.clj:944)
>> 	at clojure.core.async.impl.ioc_macros$take_BANG_$fn__13264.invoke(ioc_macros.clj:953)
>> 	at clojure.core.async.impl.channels.ManyToManyChannel$fn__8952.invoke(channels.clj:133)
>> 	at clojure.lang.AFn.run(AFn.java:24)
>> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>> 	at java.lang.Thread.run(Thread.java:744)
>> Caused by: org.jclouds.http.HttpResponseException: request: HEAD https://phalliday-opscenter-backups-4.s3.amazonaws.com/snapshots/728139a7-093b-40f7-96a9-767a178fa5a5/sstables/OpsCenter-rollups300-jb-96-Filter.db
<https://phalliday-opscenter-backups-4.s3.amazonaws.com/snapshots/728139a7-093b-40f7-96a9-767a178fa5a5/sstables/OpsCenter-rollups300-jb-96-Filter.db>
HTTP/1.1 failed with response: HTTP/1.1 403 Forbidden
>> 	at org.jclouds.aws.handlers.ParseAWSErrorFromXmlContent.handleError(ParseAWSErrorFromXmlContent.java:63)
>> 	... 25 more
>> DEBUG [clojure-agent-send-off-pool-9] 2014-12-05 16:29:15,601 Sending message '["172.31.10.206",{"details":{"error":"HEAD
https:\/\/phalliday-opscenter-backups-4.s3.amazonaws.com <http://phalliday-opscenter-backups-4.s3.amazonaws.com/>\/snapshots\/728139a7-093b-40f7-96a9-767a178fa5a5\/sstables\/OpsCenter-rollups300-jb-96-Filter.db
HTTP\/1.1 -> HTTP\/1.1 403 Forbidden","filename":"OpsCenter-rollups300-jb-96-Filter.db","keyspace":"OpsCenter","cf":"rollups300"},"state":"running","type":"sstable","destination_id":"380a2db2ac2042c5bd43043d8ae53b6f","parent_req":"eb335075-765a-46b0-bcb8-89188d93cb1d"}]'
to channel '/snapshot/status'.
>> ERROR [async-dispatch-31] 2014-12-05 16:29:15,696 Got exception HEAD https://phalliday-opscenter-backups-4.s3.amazonaws.com/snapshots/728139a7-093b-40f7-96a9-767a178fa5a5/sstables/OpsCenter-pdps-jb-114-CompressionInfo.db
<https://phalliday-opscenter-backups-4.s3.amazonaws.com/snapshots/728139a7-093b-40f7-96a9-767a178fa5a5/sstables/OpsCenter-pdps-jb-114-CompressionInfo.db>
HTTP/1.1 -> HTTP/1.1 403 Forbidden while sending opsagent.backups.entities.SSTable@a4c93
to :380a2db2ac2042c5bd43043d8ae53b6f
>> org.jclouds.rest.AuthorizationException: HEAD https://phalliday-opscenter-backups-4.s3.amazonaws.com/snapshots/728139a7-093b-40f7-96a9-767a178fa5a5/sstables/OpsCenter-pdps-jb-114-CompressionInfo.db
<https://phalliday-opscenter-backups-4.s3.amazonaws.com/snapshots/728139a7-093b-40f7-96a9-767a178fa5a5/sstables/OpsCenter-pdps-jb-114-CompressionInfo.db>
HTTP/1.1 -> HTTP/1.1 403 Forbidden
>> 	at org.jclouds.aws.handlers.ParseAWSErrorFromXmlContent.refineException(ParseAWSErrorFromXmlContent.java:122)
>> 	at org.jclouds.s3.handlers.ParseS3ErrorFromXmlContent.refineException(ParseS3ErrorFromXmlContent.java:94)
>> 	at org.jclouds.aws.handlers.ParseAWSErrorFromXmlContent.handleError(ParseAWSErrorFromXmlContent.java:89)
>> 	at org.jclouds.http.handlers.DelegatingErrorHandler.handleError(DelegatingErrorHandler.java:65)
>> 	at org.jclouds.http.internal.BaseHttpCommandExecutorService.shouldContinue(BaseHttpCommandExecutorService.java:135)
>> 	at org.jclouds.http.internal.BaseHttpCommandExecutorService.invoke(BaseHttpCommandExecutorService.java:105)
>> 	at org.jclouds.rest.internal.InvokeSyncToAsyncHttpMethod.invoke(InvokeSyncToAsyncHttpMethod.java:128)
>> 	at org.jclouds.rest.internal.InvokeSyncToAsyncHttpMethod.apply(InvokeSyncToAsyncHttpMethod.java:94)
>> 	at org.jclouds.rest.internal.InvokeSyncToAsyncHttpMethod.apply(InvokeSyncToAsyncHttpMethod.java:55)
>> 	at org.jclouds.rest.internal.DelegatesToInvocationFunction.handle(DelegatesToInvocationFunction.java:156)
>> 	at org.jclouds.rest.internal.DelegatesToInvocationFunction.invoke(DelegatesToInvocationFunction.java:123)
>> 	at com.sun.proxy.$Proxy48.objectExists(Unknown Source)
>> 	at org.jclouds.s3.blobstore.S3BlobStore.blobExists(S3BlobStore.java:173)
>> 	at org.jclouds.blobstore2$blob_exists_QMARK_.invoke(blobstore2.clj:238)
>> 	at opsagent.backups.destinations$create_blob.invoke(destinations.clj:43)
>> 	at opsagent.backups.destinations$eval15431$fn__15460.invoke(destinations.clj:123)
>> 	at opsagent.backups.destinations$eval15244$fn__15245$G__15235__15254.invoke(destinations.clj:19)
>> 	at opsagent.backups.staging$start_staging_BANG_$fn__15649$state_machine__13248__auto____15650$fn__15655$fn__15683.invoke(staging.clj:46)
>> 	at opsagent.backups.staging$start_staging_BANG_$fn__15649$state_machine__13248__auto____15650$fn__15655.invoke(staging.clj:44)
>> 	at opsagent.backups.staging$start_staging_BANG_$fn__15649$state_machine__13248__auto____15650.invoke(staging.clj:40)
>> 	at clojure.core.async.impl.ioc_macros$run_state_machine.invoke(ioc_macros.clj:940)
>> 	at clojure.core.async.impl.ioc_macros$run_state_machine_wrapped.invoke(ioc_macros.clj:944)
>> 	at clojure.core.async.impl.ioc_macros$take_BANG_$fn__13264.invoke(ioc_macros.clj:953)
>> 	at clojure.core.async.impl.channels.ManyToManyChannel$fn__8952.invoke(channels.clj:133)
>> 	at clojure.lang.AFn.run(AFn.java:24)
>> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>> 	at java.lang.Thread.run(Thread.java:744)
>> Caused by: org.jclouds.http.HttpResponseException: request: HEAD https://phalliday-opscenter-backups-4.s3.amazonaws.com/snapshots/728139a7-093b-40f7-96a9-767a178fa5a5/sstables/OpsCenter-pdps-jb-114-CompressionInfo.db
<https://phalliday-opscenter-backups-4.s3.amazonaws.com/snapshots/728139a7-093b-40f7-96a9-767a178fa5a5/sstables/OpsCenter-pdps-jb-114-CompressionInfo.db>
HTTP/1.1 failed with response: HTTP/1.1 403 Forbidden
>> 	at org.jclouds.aws.handlers.ParseAWSErrorFromXmlContent.handleError(ParseAWSErrorFromXmlContent.java:63)
>> 	... 25 more
>> DEBUG [clojure-agent-send-off-pool-9] 2014-12-05 16:29:15,697 Sending message '["172.31.10.206",{"details":{"error":"HEAD
https:\/\/phalliday-opscenter-backups-4.s3.amazonaws.com <http://phalliday-opscenter-backups-4.s3.amazonaws.com/>\/snapshots\/728139a7-093b-40f7-96a9-767a178fa5a5\/sstables\/OpsCenter-pdps-jb-114-CompressionInfo.db
HTTP\/1.1 -> HTTP\/1.1 403 Forbidden","filename":"OpsCenter-pdps-jb-114-CompressionInfo.db","keyspace":"OpsCenter","cf":"pdps"},"state":"running","type":"sstable","destination_id":"380a2db2ac2042c5bd43043d8ae53b6f","parent_req":"eb335075-765a-46b0-bcb8-89188d93cb1d"}]'
to channel '/snapshot/status'.
>> ERROR [async-dispatch-31] 2014-12-05 16:29:15,792 Got exception HEAD https://phalliday-opscenter-backups-4.s3.amazonaws.com/snapshots/728139a7-093b-40f7-96a9-767a178fa5a5/sstables/OpsCenter-pdps-jb-114-Statistics.db
<https://phalliday-opscenter-backups-4.s3.amazonaws.com/snapshots/728139a7-093b-40f7-96a9-767a178fa5a5/sstables/OpsCenter-pdps-jb-114-Statistics.db>
HTTP/1.1 -> HTTP/1.1 403 Forbidden while sending opsagent.backups.entities.SSTable@44e5d97
to :380a2db2ac2042c5bd43043d8ae53b6f
>> org.jclouds.rest.AuthorizationException: HEAD https://phalliday-opscenter-backups-4.s3.amazonaws.com/snapshots/728139a7-093b-40f7-96a9-767a178fa5a5/sstables/OpsCenter-pdps-jb-114-Statistics.db
<https://phalliday-opscenter-backups-4.s3.amazonaws.com/snapshots/728139a7-093b-40f7-96a9-767a178fa5a5/sstables/OpsCenter-pdps-jb-114-Statistics.db>
HTTP/1.1 -> HTTP/1.1 403 Forbidden
>> 	at org.jclouds.aws.handlers.ParseAWSErrorFromXmlContent.refineException(ParseAWSErrorFromXmlContent.java:122)
>> 	at org.jclouds.s3.handlers.ParseS3ErrorFromXmlContent.refineException(ParseS3ErrorFromXmlContent.java:94)
>> 	at org.jclouds.aws.handlers.ParseAWSErrorFromXmlContent.handleError(ParseAWSErrorFromXmlContent.java:89)
>> 	at org.jclouds.http.handlers.DelegatingErrorHandler.handleError(DelegatingErrorHandler.java:65)
>> 	at org.jclouds.http.internal.BaseHttpCommandExecutorService.shouldContinue(BaseHttpCommandExecutorService.java:135)
>> 	at org.jclouds.http.internal.BaseHttpCommandExecutorService.invoke(BaseHttpCommandExecutorService.java:105)
>> 	at org.jclouds.rest.internal.InvokeSyncToAsyncHttpMethod.invoke(InvokeSyncToAsyncHttpMethod.java:128)
>> 	at org.jclouds.rest.internal.InvokeSyncToAsyncHttpMethod.apply(InvokeSyncToAsyncHttpMethod.java:94)
>> 	at org.jclouds.rest.internal.InvokeSyncToAsyncHttpMethod.apply(InvokeSyncToAsyncHttpMethod.java:55)
>> 	at org.jclouds.rest.internal.DelegatesToInvocationFunction.handle(DelegatesToInvocationFunction.java:156)
>> 	at org.jclouds.rest.internal.DelegatesToInvocationFunction.invoke(DelegatesToInvocationFunction.java:123)
>> 	at com.sun.proxy.$Proxy48.objectExists(Unknown Source)
>> 	at org.jclouds.s3.blobstore.S3BlobStore.blobExists(S3BlobStore.java:173)
>> 	at org.jclouds.blobstore2$blob_exists_QMARK_.invoke(blobstore2.clj:238)
>> 	at opsagent.backups.destinations$create_blob.invoke(destinations.clj:43)
>> 	at opsagent.backups.destinations$eval15431$fn__15460.invoke(destinations.clj:123)
>> 	at opsagent.backups.destinations$eval15244$fn__15245$G__15235__15254.invoke(destinations.clj:19)
>> 	at opsagent.backups.staging$start_staging_BANG_$fn__15649$state_machine__13248__auto____15650$fn__15655$fn__15683.invoke(staging.clj:46)
>> 	at opsagent.backups.staging$start_staging_BANG_$fn__15649$state_machine__13248__auto____15650$fn__15655.invoke(staging.clj:44)
>> 	at opsagent.backups.staging$start_staging_BANG_$fn__15649$state_machine__13248__auto____15650.invoke(staging.clj:40)
>> 	at clojure.core.async.impl.ioc_macros$run_state_machine.invoke(ioc_macros.clj:940)
>> 	at clojure.core.async.impl.ioc_macros$run_state_machine_wrapped.invoke(ioc_macros.clj:944)
>> 	at clojure.core.async.impl.ioc_macros$take_BANG_$fn__13264.invoke(ioc_macros.clj:953)
>> 	at clojure.core.async.impl.channels.ManyToManyChannel$fn__8952.invoke(channels.clj:133)
>> 	at clojure.lang.AFn.run(AFn.java:24)
>> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>> 	at java.lang.Thread.run(Thread.java:744)
>> Caused by: org.jclouds.http.HttpResponseException: request: HEAD https://phalliday-opscenter-backups-4.s3.amazonaws.com/snapshots/728139a7-093b-40f7-96a9-767a178fa5a5/sstables/OpsCenter-pdps-jb-114-Statistics.db
<https://phalliday-opscenter-backups-4.s3.amazonaws.com/snapshots/728139a7-093b-40f7-96a9-767a178fa5a5/sstables/OpsCenter-pdps-jb-114-Statistics.db>
HTTP/1.1 failed with response: HTTP/1.1 403 Forbidden
>> 	at org.jclouds.aws.handlers.ParseAWSErrorFromXmlContent.handleError(ParseAWSErrorFromXmlContent.java:63)
>> 	... 25 more
>> DEBUG [clojure-agent-send-off-pool-9] 2014-12-05 16:29:15,793 Sending message '["172.31.10.206",{"details":{"error":"HEAD
https:\/\/phalliday-opscenter-backups-4.s3.amazonaws.com <http://phalliday-opscenter-backups-4.s3.amazonaws.com/>\/snapshots\/728139a7-093b-40f7-96a9-767a178fa5a5\/sstables\/OpsCenter-pdps-jb-114-Statistics.db
HTTP\/1.1 -> HTTP\/1.1 403 Forbidden","filename":"OpsCenter-pdps-jb-114-Statistics.db","keyspace":"OpsCenter","cf":"pdps"},"state":"running","type":"sstable","destination_id":"380a2db2ac2042c5bd43043d8ae53b6f","parent_req":"eb335075-765a-46b0-bcb8-89188d93cb1d"}]'
to channel '/snapshot/status'.
>> ERROR [async-dispatch-31] 2014-12-05 16:29:15,888 Got exception HEAD https://phalliday-opscenter-backups-4.s3.amazonaws.com/snapshots/728139a7-093b-40f7-96a9-767a178fa5a5/sstables/OpsCenter-pdps-jb-114-Data.db
<https://phalliday-opscenter-backups-4.s3.amazonaws.com/snapshots/728139a7-093b-40f7-96a9-767a178fa5a5/sstables/OpsCenter-pdps-jb-114-Data.db>
HTTP/1.1 -> HTTP/1.1 403 Forbidden while sending opsagent.backups.entities.SSTable@8ca4473b
to :380a2db2ac2042c5bd43043d8ae53b6f
>> org.jclouds.rest.AuthorizationException: HEAD https://phalliday-opscenter-backups-4.s3.amazonaws.com/snapshots/728139a7-093b-40f7-96a9-767a178fa5a5/sstables/OpsCenter-pdps-jb-114-Data.db
<https://phalliday-opscenter-backups-4.s3.amazonaws.com/snapshots/728139a7-093b-40f7-96a9-767a178fa5a5/sstables/OpsCenter-pdps-jb-114-Data.db>
HTTP/1.1 -> HTTP/1.1 403 Forbidden
>> 	at org.jclouds.aws.handlers.ParseAWSErrorFromXmlContent.refineException(ParseAWSErrorFromXmlContent.java:122)
>> 	at org.jclouds.s3.handlers.ParseS3ErrorFromXmlContent.refineException(ParseS3ErrorFromXmlContent.java:94)
>> 	at org.jclouds.aws.handlers.ParseAWSErrorFromXmlContent.handleError(ParseAWSErrorFromXmlContent.java:89)
>> 	at org.jclouds.http.handlers.DelegatingErrorHandler.handleError(DelegatingErrorHandler.java:65)
>> 	at org.jclouds.http.internal.BaseHttpCommandExecutorService.shouldContinue(BaseHttpCommandExecutorService.java:135)
>> 	at org.jclouds.http.internal.BaseHttpCommandExecutorService.invoke(BaseHttpCommandExecutorService.java:105)
>> 	at org.jclouds.rest.internal.InvokeSyncToAsyncHttpMethod.invoke(InvokeSyncToAsyncHttpMethod.java:128)
>> 	at org.jclouds.rest.internal.InvokeSyncToAsyncHttpMethod.apply(InvokeSyncToAsyncHttpMethod.java:94)
>> 	at org.jclouds.rest.internal.InvokeSyncToAsyncHttpMethod.apply(InvokeSyncToAsyncHttpMethod.java:55)
>> 	at org.jclouds.rest.internal.DelegatesToInvocationFunction.handle(DelegatesToInvocationFunction.java:156)
>> 	at org.jclouds.rest.internal.DelegatesToInvocationFunction.invoke(DelegatesToInvocationFunction.java:123)
>> 	at com.sun.proxy.$Proxy48.objectExists(Unknown Source)
>> 	at org.jclouds.s3.blobstore.S3BlobStore.blobExists(S3BlobStore.java:173)
>> 	at org.jclouds.blobstore2$blob_exists_QMARK_.invoke(blobstore2.clj:238)
>> 	at opsagent.backups.destinations$create_blob.invoke(destinations.clj:43)
>> 	at opsagent.backups.destinations$eval15431$fn__15460.invoke(destinations.clj:123)
>> 	at opsagent.backups.destinations$eval15244$fn__15245$G__15235__15254.invoke(destinations.clj:19)
>> 	at opsagent.backups.staging$start_staging_BANG_$fn__15649$state_machine__13248__auto____15650$fn__15655$fn__15683.invoke(staging.clj:46)
>> 	at opsagent.backups.staging$start_staging_BANG_$fn__15649$state_machine__13248__auto____15650$fn__15655.invoke(staging.clj:44)
>> 	at opsagent.backups.staging$start_staging_BANG_$fn__15649$state_machine__13248__auto____15650.invoke(staging.clj:40)
>> 	at clojure.core.async.impl.ioc_macros$run_state_machine.invoke(ioc_macros.clj:940)
>> 	at clojure.core.async.impl.ioc_macros$run_state_machine_wrapped.invoke(ioc_macros.clj:944)
>> 	at clojure.core.async.impl.ioc_macros$take_BANG_$fn__13264.invoke(ioc_macros.clj:953)
>> 	at clojure.core.async.impl.channels.ManyToManyChannel$fn__8952.invoke(channels.clj:133)
>> 	at clojure.lang.AFn.run(AFn.java:24)
>> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>> 	at java.lang.Thread.run(Thread.java:744)
>> Caused by: org.jclouds.http.HttpResponseException: request: HEAD https://phalliday-opscenter-backups-4.s3.amazonaws.com/snapshots/728139a7-093b-40f7-96a9-767a178fa5a5/sstables/OpsCenter-pdps-jb-114-Data.db
<https://phalliday-opscenter-backups-4.s3.amazonaws.com/snapshots/728139a7-093b-40f7-96a9-767a178fa5a5/sstables/OpsCenter-pdps-jb-114-Data.db>
HTTP/1.1 failed with response: HTTP/1.1 403 Forbidden
>> 	at org.jclouds.aws.handlers.ParseAWSErrorFromXmlContent.handleError(ParseAWSErrorFromXmlContent.java:63)
>> 	... 25 more
>> DEBUG [clojure-agent-send-off-pool-9] 2014-12-05 16:29:15,890 Sending message '["172.31.10.206",{"details":{"error":"HEAD
https:\/\/phalliday-opscenter-backups-4.s3.amazonaws.com <http://phalliday-opscenter-backups-4.s3.amazonaws.com/>\/snapshots\/728139a7-093b-40f7-96a9-767a178fa5a5\/sstables\/OpsCenter-pdps-jb-114-Data.db
HTTP\/1.1 -> HTTP\/1.1 403 Forbidden","filename":"OpsCenter-pdps-jb-114-Data.db","keyspace":"OpsCenter","cf":"pdps"},"state":"running","type":"sstable","destination_id":"380a2db2ac2042c5bd43043d8ae53b6f","parent_req":"eb335075-765a-46b0-bcb8-89188d93cb1d"}]'
to channel '/snapshot/status'.
>> DEBUG [node-details-1] 2014-12-05 16:29:16,030 Starting real-time collection
>> DEBUG [node-details-1] 2014-12-05 16:29:16,030 Collecting real-time data
>> DEBUG [clojure-agent-send-off-pool-9] 2014-12-05 16:29:16,034 Sending message '["172.31.10.206",{"os-load":0.07}]'
to channel '/node-details'.
>> DEBUG [clojure-agent-send-off-pool-9] 2014-12-05 16:29:16,298 Sending message '["172.31.10.206",{"destination_id":"380a2db2ac2042c5bd43043d8ae53b6f","type":"schema","state":"running","details":{"keyspace":"OpsCenter","filename":"OpsCenter\/schema.json"},"parent_req":"eb335075-765a-46b0-bcb8-89188d93cb1d"}]'
to channel '/snapshot/status'.
>> DEBUG [async-dispatch-31] 2014-12-05 16:29:16,402 Uploading file: snapshots/728139a7-093b-40f7-96a9-767a178fa5a5/sstables/OpsCenter-pdps-jb-115-Summary.db
>> DEBUG [async-dispatch-31] 2014-12-05 16:29:16,404 Throttle value requested is out
of range, using default.
>> ERROR [async-dispatch-31] 2014-12-05 16:29:16,510 Dec 05, 2014 4:29:16 PM org.jclouds.logging.jdk.JDKLogger
logError
>> SEVERE: Cannot retry after server error, command is not replayable: [method=org.jclouds.aws.s3.AWSS3AsyncClient.public
abstract com.google.common.util.concurrent.ListenableFuture org.jclouds.s3.S3AsyncClient.putObject(java.lang.String,org.jclouds.s3.domain.S3Object,org.jclouds.s3.options.PutObjectOptions[])[phalliday-opscenter-backups-4,
[metadata=[key=snapshots/728139a7-093b-40f7-96a9-767a178fa5a5/sstables/OpsCenter-pdps-jb-115-Summary.db,
bucket=null, uri=null, eTag=null, cacheControl=null, contentMetadata=[contentDisposition=null,
contentEncoding=null, contentLanguage=null, contentLength=107, contentMD5=null, contentType=application/octet-stream,
expires=null], lastModified=null, owner=null, storageClass=STANDARD, userMetadata={}]], [Lorg.jclouds.s3.options.PutObjectOptions;@4e0ecbfe],
request=PUT https://phalliday-opscenter-backups-4.s3.amazonaws.com/snapshots/728139a7-093b-40f7-96a9-767a178fa5a5/sstables/OpsCenter-pdps-jb-115-Summary.db
<https://phalliday-opscenter-backups-4.s3.amazonaws.com/snapshots/728139a7-093b-40f7-96a9-767a178fa5a5/sstables/OpsCenter-pdps-jb-115-Summary.db>
HTTP/1.1]
>> 
>> DEBUG [node-details-1] 2014-12-05 16:29:21,034 Starting real-time collection
>> DEBUG [node-details-1] 2014-12-05 16:29:21,035 Collecting real-time data
>> DEBUG [node-details-1] 2014-12-05 16:29:21,038 Send requested, but nothing new to
send
>> 
>> 
>> 
>> 
>> 
>> <http://www.datastax.com/ <http://www.datastax.com/>>
>> PETER HALLIDAY
>> OpsCenter | Sr Software Engineer | 607.329.6905 | peter@datastax.com <mailto:peter@datastax.com>
>> 
>> <https://twitter.com/phalliday <https://twitter.com/phalliday>>  <https://www.linkedin.com/profile/view?id=7030852
<https://www.linkedin.com/profile/view?id=7030852>>
> 
> -- 
> Andrew Gaul
> http://gaul.org/ <http://gaul.org/>

Mime
View raw message