jclouds-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Peter Halliday <pe...@datastax.com>
Subject Thread being killed
Date Fri, 05 Dec 2014 16:41:48 GMT
I’m using JClouds BlobStore in an application that’s using it to send files to S3.  It
does this in batches like after a snapshot is run on a database.  And then all of the files
are transferred to S3.  We were testing error states.  And we were testing changing a User’s
Access credentials from inactive to active and back.  When we go from active to inactive credentials
during this transfer to S3.  Our thread gets killed.  The last log entry is from JClouds.
 Our code is literally catching all errors, and so nothing is being thrown at that time. 
Our application is in clojure.  You can see in the log, that we upload successfully fine,
and then after we switch to invalid we “correct” get errors that get returned to our application.
 However, then we hit the SEVERE log line, and then our thread is dead and never comes back
:-( 



DEBUG [clojure-agent-send-off-pool-9] 2014-12-05 16:29:14,398 Sending message '["172.31.10.206",{"destination_id":"380a2db2ac2042c5bd43043d8ae53b6f","type":"sstable","state":"running","details":{"filename":"OpsCenter-rollups300-jb-96-Data.db","keyspace":"OpsCenter","cf":"rollups300"},"parent_req":"eb335075-765a-46b0-bcb8-89188d93cb1d"}]'
to channel '/snapshot/status'.
DEBUG [async-dispatch-31] 2014-12-05 16:29:14,496 Uploading file: snapshots/728139a7-093b-40f7-96a9-767a178fa5a5/sstables/OpsCenter-rollups300-jb-98-CompressionInfo.db
DEBUG [async-dispatch-31] 2014-12-05 16:29:14,498 Throttle value requested is out of range,
using default.
DEBUG [clojure-agent-send-off-pool-9] 2014-12-05 16:29:14,695 Sending message '["172.31.10.206",{"destination_id":"380a2db2ac2042c5bd43043d8ae53b6f","type":"sstable","state":"running","details":{"filename":"OpsCenter-rollups300-jb-98-CompressionInfo.db","keyspace":"OpsCenter","cf":"rollups300"},"parent_req":"eb335075-765a-46b0-bcb8-89188d93cb1d"}]'
to channel '/snapshot/status'.
DEBUG [async-dispatch-31] 2014-12-05 16:29:14,797 Uploading file: snapshots/728139a7-093b-40f7-96a9-767a178fa5a5/sstables/OpsCenter-rollups300-jb-98-Index.db
DEBUG [async-dispatch-31] 2014-12-05 16:29:14,809 Throttle value requested is out of range,
using default.
DEBUG [clojure-agent-send-off-pool-9] 2014-12-05 16:29:15,105 Sending message '["172.31.10.206",{"destination_id":"380a2db2ac2042c5bd43043d8ae53b6f","type":"sstable","state":"running","details":{"filename":"OpsCenter-rollups300-jb-98-Index.db","keyspace":"OpsCenter","cf":"rollups300"},"parent_req":"eb335075-765a-46b0-bcb8-89188d93cb1d"}]'
to channel '/snapshot/status'.
DEBUG [clojure-agent-send-off-pool-9] 2014-12-05 16:29:15,204 Sending message '["172.31.10.206",{"destination_id":"380a2db2ac2042c5bd43043d8ae53b6f","type":"sstable","state":"running","details":{"filename":"OpsCenter-rollups300-jb-97-Statistics.db","keyspace":"OpsCenter","cf":"rollups300"},"parent_req":"eb335075-765a-46b0-bcb8-89188d93cb1d"}]'
to channel '/snapshot/status'.
DEBUG [clojure-agent-send-off-pool-9] 2014-12-05 16:29:15,305 Sending message '["172.31.10.206",{"destination_id":"380a2db2ac2042c5bd43043d8ae53b6f","type":"sstable","state":"running","details":{"filename":"OpsCenter-rollups300-jb-96-Index.db","keyspace":"OpsCenter","cf":"rollups300"},"parent_req":"eb335075-765a-46b0-bcb8-89188d93cb1d"}]'
to channel '/snapshot/status'.
DEBUG [clojure-agent-send-off-pool-9] 2014-12-05 16:29:15,406 Sending message '["172.31.10.206",{"destination_id":"380a2db2ac2042c5bd43043d8ae53b6f","type":"sstable","state":"running","details":{"filename":"OpsCenter-rollups300-jb-97-Summary.db","keyspace":"OpsCenter","cf":"rollups300"},"parent_req":"eb335075-765a-46b0-bcb8-89188d93cb1d"}]'
to channel '/snapshot/status'.
ERROR [async-dispatch-31] 2014-12-05 16:29:15,503 Got exception HEAD https://phalliday-opscenter-backups-4.s3.amazonaws.com/snapshots/728139a7-093b-40f7-96a9-767a178fa5a5/sstables/OpsCenter-rollups300-jb-96-Summary.db
HTTP/1.1 -> HTTP/1.1 403 Forbidden while sending opsagent.backups.entities.SSTable@48fee98c
to :380a2db2ac2042c5bd43043d8ae53b6f
org.jclouds.rest.AuthorizationException: HEAD https://phalliday-opscenter-backups-4.s3.amazonaws.com/snapshots/728139a7-093b-40f7-96a9-767a178fa5a5/sstables/OpsCenter-rollups300-jb-96-Summary.db
HTTP/1.1 -> HTTP/1.1 403 Forbidden
	at org.jclouds.aws.handlers.ParseAWSErrorFromXmlContent.refineException(ParseAWSErrorFromXmlContent.java:122)
	at org.jclouds.s3.handlers.ParseS3ErrorFromXmlContent.refineException(ParseS3ErrorFromXmlContent.java:94)
	at org.jclouds.aws.handlers.ParseAWSErrorFromXmlContent.handleError(ParseAWSErrorFromXmlContent.java:89)
	at org.jclouds.http.handlers.DelegatingErrorHandler.handleError(DelegatingErrorHandler.java:65)
	at org.jclouds.http.internal.BaseHttpCommandExecutorService.shouldContinue(BaseHttpCommandExecutorService.java:135)
	at org.jclouds.http.internal.BaseHttpCommandExecutorService.invoke(BaseHttpCommandExecutorService.java:105)
	at org.jclouds.rest.internal.InvokeSyncToAsyncHttpMethod.invoke(InvokeSyncToAsyncHttpMethod.java:128)
	at org.jclouds.rest.internal.InvokeSyncToAsyncHttpMethod.apply(InvokeSyncToAsyncHttpMethod.java:94)
	at org.jclouds.rest.internal.InvokeSyncToAsyncHttpMethod.apply(InvokeSyncToAsyncHttpMethod.java:55)
	at org.jclouds.rest.internal.DelegatesToInvocationFunction.handle(DelegatesToInvocationFunction.java:156)
	at org.jclouds.rest.internal.DelegatesToInvocationFunction.invoke(DelegatesToInvocationFunction.java:123)
	at com.sun.proxy.$Proxy48.objectExists(Unknown Source)
	at org.jclouds.s3.blobstore.S3BlobStore.blobExists(S3BlobStore.java:173)
	at org.jclouds.blobstore2$blob_exists_QMARK_.invoke(blobstore2.clj:238)
	at opsagent.backups.destinations$create_blob.invoke(destinations.clj:43)
	at opsagent.backups.destinations$eval15431$fn__15460.invoke(destinations.clj:123)
	at opsagent.backups.destinations$eval15244$fn__15245$G__15235__15254.invoke(destinations.clj:19)
	at opsagent.backups.staging$start_staging_BANG_$fn__15649$state_machine__13248__auto____15650$fn__15655$fn__15683.invoke(staging.clj:46)
	at opsagent.backups.staging$start_staging_BANG_$fn__15649$state_machine__13248__auto____15650$fn__15655.invoke(staging.clj:44)
	at opsagent.backups.staging$start_staging_BANG_$fn__15649$state_machine__13248__auto____15650.invoke(staging.clj:40)
	at clojure.core.async.impl.ioc_macros$run_state_machine.invoke(ioc_macros.clj:940)
	at clojure.core.async.impl.ioc_macros$run_state_machine_wrapped.invoke(ioc_macros.clj:944)
	at clojure.core.async.impl.ioc_macros$take_BANG_$fn__13264.invoke(ioc_macros.clj:953)
	at clojure.core.async.impl.channels.ManyToManyChannel$fn__8952.invoke(channels.clj:133)
	at clojure.lang.AFn.run(AFn.java:24)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:744)
Caused by: org.jclouds.http.HttpResponseException: request: HEAD https://phalliday-opscenter-backups-4.s3.amazonaws.com/snapshots/728139a7-093b-40f7-96a9-767a178fa5a5/sstables/OpsCenter-rollups300-jb-96-Summary.db
HTTP/1.1 failed with response: HTTP/1.1 403 Forbidden
	at org.jclouds.aws.handlers.ParseAWSErrorFromXmlContent.handleError(ParseAWSErrorFromXmlContent.java:63)
	... 25 more
DEBUG [clojure-agent-send-off-pool-9] 2014-12-05 16:29:15,504 Sending message '["172.31.10.206",{"details":{"error":"HEAD
https:\/\/phalliday-opscenter-backups-4.s3.amazonaws.com\/snapshots\/728139a7-093b-40f7-96a9-767a178fa5a5\/sstables\/OpsCenter-rollups300-jb-96-Summary.db
HTTP\/1.1 -> HTTP\/1.1 403 Forbidden","filename":"OpsCenter-rollups300-jb-96-Summary.db","keyspace":"OpsCenter","cf":"rollups300"},"state":"running","type":"sstable","destination_id":"380a2db2ac2042c5bd43043d8ae53b6f","parent_req":"eb335075-765a-46b0-bcb8-89188d93cb1d"}]'
to channel '/snapshot/status'.
ERROR [async-dispatch-31] 2014-12-05 16:29:15,600 Got exception HEAD https://phalliday-opscenter-backups-4.s3.amazonaws.com/snapshots/728139a7-093b-40f7-96a9-767a178fa5a5/sstables/OpsCenter-rollups300-jb-96-Filter.db
HTTP/1.1 -> HTTP/1.1 403 Forbidden while sending opsagent.backups.entities.SSTable@91afeb42
to :380a2db2ac2042c5bd43043d8ae53b6f
org.jclouds.rest.AuthorizationException: HEAD https://phalliday-opscenter-backups-4.s3.amazonaws.com/snapshots/728139a7-093b-40f7-96a9-767a178fa5a5/sstables/OpsCenter-rollups300-jb-96-Filter.db
HTTP/1.1 -> HTTP/1.1 403 Forbidden
	at org.jclouds.aws.handlers.ParseAWSErrorFromXmlContent.refineException(ParseAWSErrorFromXmlContent.java:122)
	at org.jclouds.s3.handlers.ParseS3ErrorFromXmlContent.refineException(ParseS3ErrorFromXmlContent.java:94)
	at org.jclouds.aws.handlers.ParseAWSErrorFromXmlContent.handleError(ParseAWSErrorFromXmlContent.java:89)
	at org.jclouds.http.handlers.DelegatingErrorHandler.handleError(DelegatingErrorHandler.java:65)
	at org.jclouds.http.internal.BaseHttpCommandExecutorService.shouldContinue(BaseHttpCommandExecutorService.java:135)
	at org.jclouds.http.internal.BaseHttpCommandExecutorService.invoke(BaseHttpCommandExecutorService.java:105)
	at org.jclouds.rest.internal.InvokeSyncToAsyncHttpMethod.invoke(InvokeSyncToAsyncHttpMethod.java:128)
	at org.jclouds.rest.internal.InvokeSyncToAsyncHttpMethod.apply(InvokeSyncToAsyncHttpMethod.java:94)
	at org.jclouds.rest.internal.InvokeSyncToAsyncHttpMethod.apply(InvokeSyncToAsyncHttpMethod.java:55)
	at org.jclouds.rest.internal.DelegatesToInvocationFunction.handle(DelegatesToInvocationFunction.java:156)
	at org.jclouds.rest.internal.DelegatesToInvocationFunction.invoke(DelegatesToInvocationFunction.java:123)
	at com.sun.proxy.$Proxy48.objectExists(Unknown Source)
	at org.jclouds.s3.blobstore.S3BlobStore.blobExists(S3BlobStore.java:173)
	at org.jclouds.blobstore2$blob_exists_QMARK_.invoke(blobstore2.clj:238)
	at opsagent.backups.destinations$create_blob.invoke(destinations.clj:43)
	at opsagent.backups.destinations$eval15431$fn__15460.invoke(destinations.clj:123)
	at opsagent.backups.destinations$eval15244$fn__15245$G__15235__15254.invoke(destinations.clj:19)
	at opsagent.backups.staging$start_staging_BANG_$fn__15649$state_machine__13248__auto____15650$fn__15655$fn__15683.invoke(staging.clj:46)
	at opsagent.backups.staging$start_staging_BANG_$fn__15649$state_machine__13248__auto____15650$fn__15655.invoke(staging.clj:44)
	at opsagent.backups.staging$start_staging_BANG_$fn__15649$state_machine__13248__auto____15650.invoke(staging.clj:40)
	at clojure.core.async.impl.ioc_macros$run_state_machine.invoke(ioc_macros.clj:940)
	at clojure.core.async.impl.ioc_macros$run_state_machine_wrapped.invoke(ioc_macros.clj:944)
	at clojure.core.async.impl.ioc_macros$take_BANG_$fn__13264.invoke(ioc_macros.clj:953)
	at clojure.core.async.impl.channels.ManyToManyChannel$fn__8952.invoke(channels.clj:133)
	at clojure.lang.AFn.run(AFn.java:24)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:744)
Caused by: org.jclouds.http.HttpResponseException: request: HEAD https://phalliday-opscenter-backups-4.s3.amazonaws.com/snapshots/728139a7-093b-40f7-96a9-767a178fa5a5/sstables/OpsCenter-rollups300-jb-96-Filter.db
HTTP/1.1 failed with response: HTTP/1.1 403 Forbidden
	at org.jclouds.aws.handlers.ParseAWSErrorFromXmlContent.handleError(ParseAWSErrorFromXmlContent.java:63)
	... 25 more
DEBUG [clojure-agent-send-off-pool-9] 2014-12-05 16:29:15,601 Sending message '["172.31.10.206",{"details":{"error":"HEAD
https:\/\/phalliday-opscenter-backups-4.s3.amazonaws.com\/snapshots\/728139a7-093b-40f7-96a9-767a178fa5a5\/sstables\/OpsCenter-rollups300-jb-96-Filter.db
HTTP\/1.1 -> HTTP\/1.1 403 Forbidden","filename":"OpsCenter-rollups300-jb-96-Filter.db","keyspace":"OpsCenter","cf":"rollups300"},"state":"running","type":"sstable","destination_id":"380a2db2ac2042c5bd43043d8ae53b6f","parent_req":"eb335075-765a-46b0-bcb8-89188d93cb1d"}]'
to channel '/snapshot/status'.
ERROR [async-dispatch-31] 2014-12-05 16:29:15,696 Got exception HEAD https://phalliday-opscenter-backups-4.s3.amazonaws.com/snapshots/728139a7-093b-40f7-96a9-767a178fa5a5/sstables/OpsCenter-pdps-jb-114-CompressionInfo.db
HTTP/1.1 -> HTTP/1.1 403 Forbidden while sending opsagent.backups.entities.SSTable@a4c93
to :380a2db2ac2042c5bd43043d8ae53b6f
org.jclouds.rest.AuthorizationException: HEAD https://phalliday-opscenter-backups-4.s3.amazonaws.com/snapshots/728139a7-093b-40f7-96a9-767a178fa5a5/sstables/OpsCenter-pdps-jb-114-CompressionInfo.db
HTTP/1.1 -> HTTP/1.1 403 Forbidden
	at org.jclouds.aws.handlers.ParseAWSErrorFromXmlContent.refineException(ParseAWSErrorFromXmlContent.java:122)
	at org.jclouds.s3.handlers.ParseS3ErrorFromXmlContent.refineException(ParseS3ErrorFromXmlContent.java:94)
	at org.jclouds.aws.handlers.ParseAWSErrorFromXmlContent.handleError(ParseAWSErrorFromXmlContent.java:89)
	at org.jclouds.http.handlers.DelegatingErrorHandler.handleError(DelegatingErrorHandler.java:65)
	at org.jclouds.http.internal.BaseHttpCommandExecutorService.shouldContinue(BaseHttpCommandExecutorService.java:135)
	at org.jclouds.http.internal.BaseHttpCommandExecutorService.invoke(BaseHttpCommandExecutorService.java:105)
	at org.jclouds.rest.internal.InvokeSyncToAsyncHttpMethod.invoke(InvokeSyncToAsyncHttpMethod.java:128)
	at org.jclouds.rest.internal.InvokeSyncToAsyncHttpMethod.apply(InvokeSyncToAsyncHttpMethod.java:94)
	at org.jclouds.rest.internal.InvokeSyncToAsyncHttpMethod.apply(InvokeSyncToAsyncHttpMethod.java:55)
	at org.jclouds.rest.internal.DelegatesToInvocationFunction.handle(DelegatesToInvocationFunction.java:156)
	at org.jclouds.rest.internal.DelegatesToInvocationFunction.invoke(DelegatesToInvocationFunction.java:123)
	at com.sun.proxy.$Proxy48.objectExists(Unknown Source)
	at org.jclouds.s3.blobstore.S3BlobStore.blobExists(S3BlobStore.java:173)
	at org.jclouds.blobstore2$blob_exists_QMARK_.invoke(blobstore2.clj:238)
	at opsagent.backups.destinations$create_blob.invoke(destinations.clj:43)
	at opsagent.backups.destinations$eval15431$fn__15460.invoke(destinations.clj:123)
	at opsagent.backups.destinations$eval15244$fn__15245$G__15235__15254.invoke(destinations.clj:19)
	at opsagent.backups.staging$start_staging_BANG_$fn__15649$state_machine__13248__auto____15650$fn__15655$fn__15683.invoke(staging.clj:46)
	at opsagent.backups.staging$start_staging_BANG_$fn__15649$state_machine__13248__auto____15650$fn__15655.invoke(staging.clj:44)
	at opsagent.backups.staging$start_staging_BANG_$fn__15649$state_machine__13248__auto____15650.invoke(staging.clj:40)
	at clojure.core.async.impl.ioc_macros$run_state_machine.invoke(ioc_macros.clj:940)
	at clojure.core.async.impl.ioc_macros$run_state_machine_wrapped.invoke(ioc_macros.clj:944)
	at clojure.core.async.impl.ioc_macros$take_BANG_$fn__13264.invoke(ioc_macros.clj:953)
	at clojure.core.async.impl.channels.ManyToManyChannel$fn__8952.invoke(channels.clj:133)
	at clojure.lang.AFn.run(AFn.java:24)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:744)
Caused by: org.jclouds.http.HttpResponseException: request: HEAD https://phalliday-opscenter-backups-4.s3.amazonaws.com/snapshots/728139a7-093b-40f7-96a9-767a178fa5a5/sstables/OpsCenter-pdps-jb-114-CompressionInfo.db
HTTP/1.1 failed with response: HTTP/1.1 403 Forbidden
	at org.jclouds.aws.handlers.ParseAWSErrorFromXmlContent.handleError(ParseAWSErrorFromXmlContent.java:63)
	... 25 more
DEBUG [clojure-agent-send-off-pool-9] 2014-12-05 16:29:15,697 Sending message '["172.31.10.206",{"details":{"error":"HEAD
https:\/\/phalliday-opscenter-backups-4.s3.amazonaws.com\/snapshots\/728139a7-093b-40f7-96a9-767a178fa5a5\/sstables\/OpsCenter-pdps-jb-114-CompressionInfo.db
HTTP\/1.1 -> HTTP\/1.1 403 Forbidden","filename":"OpsCenter-pdps-jb-114-CompressionInfo.db","keyspace":"OpsCenter","cf":"pdps"},"state":"running","type":"sstable","destination_id":"380a2db2ac2042c5bd43043d8ae53b6f","parent_req":"eb335075-765a-46b0-bcb8-89188d93cb1d"}]'
to channel '/snapshot/status'.
ERROR [async-dispatch-31] 2014-12-05 16:29:15,792 Got exception HEAD https://phalliday-opscenter-backups-4.s3.amazonaws.com/snapshots/728139a7-093b-40f7-96a9-767a178fa5a5/sstables/OpsCenter-pdps-jb-114-Statistics.db
HTTP/1.1 -> HTTP/1.1 403 Forbidden while sending opsagent.backups.entities.SSTable@44e5d97
to :380a2db2ac2042c5bd43043d8ae53b6f
org.jclouds.rest.AuthorizationException: HEAD https://phalliday-opscenter-backups-4.s3.amazonaws.com/snapshots/728139a7-093b-40f7-96a9-767a178fa5a5/sstables/OpsCenter-pdps-jb-114-Statistics.db
HTTP/1.1 -> HTTP/1.1 403 Forbidden
	at org.jclouds.aws.handlers.ParseAWSErrorFromXmlContent.refineException(ParseAWSErrorFromXmlContent.java:122)
	at org.jclouds.s3.handlers.ParseS3ErrorFromXmlContent.refineException(ParseS3ErrorFromXmlContent.java:94)
	at org.jclouds.aws.handlers.ParseAWSErrorFromXmlContent.handleError(ParseAWSErrorFromXmlContent.java:89)
	at org.jclouds.http.handlers.DelegatingErrorHandler.handleError(DelegatingErrorHandler.java:65)
	at org.jclouds.http.internal.BaseHttpCommandExecutorService.shouldContinue(BaseHttpCommandExecutorService.java:135)
	at org.jclouds.http.internal.BaseHttpCommandExecutorService.invoke(BaseHttpCommandExecutorService.java:105)
	at org.jclouds.rest.internal.InvokeSyncToAsyncHttpMethod.invoke(InvokeSyncToAsyncHttpMethod.java:128)
	at org.jclouds.rest.internal.InvokeSyncToAsyncHttpMethod.apply(InvokeSyncToAsyncHttpMethod.java:94)
	at org.jclouds.rest.internal.InvokeSyncToAsyncHttpMethod.apply(InvokeSyncToAsyncHttpMethod.java:55)
	at org.jclouds.rest.internal.DelegatesToInvocationFunction.handle(DelegatesToInvocationFunction.java:156)
	at org.jclouds.rest.internal.DelegatesToInvocationFunction.invoke(DelegatesToInvocationFunction.java:123)
	at com.sun.proxy.$Proxy48.objectExists(Unknown Source)
	at org.jclouds.s3.blobstore.S3BlobStore.blobExists(S3BlobStore.java:173)
	at org.jclouds.blobstore2$blob_exists_QMARK_.invoke(blobstore2.clj:238)
	at opsagent.backups.destinations$create_blob.invoke(destinations.clj:43)
	at opsagent.backups.destinations$eval15431$fn__15460.invoke(destinations.clj:123)
	at opsagent.backups.destinations$eval15244$fn__15245$G__15235__15254.invoke(destinations.clj:19)
	at opsagent.backups.staging$start_staging_BANG_$fn__15649$state_machine__13248__auto____15650$fn__15655$fn__15683.invoke(staging.clj:46)
	at opsagent.backups.staging$start_staging_BANG_$fn__15649$state_machine__13248__auto____15650$fn__15655.invoke(staging.clj:44)
	at opsagent.backups.staging$start_staging_BANG_$fn__15649$state_machine__13248__auto____15650.invoke(staging.clj:40)
	at clojure.core.async.impl.ioc_macros$run_state_machine.invoke(ioc_macros.clj:940)
	at clojure.core.async.impl.ioc_macros$run_state_machine_wrapped.invoke(ioc_macros.clj:944)
	at clojure.core.async.impl.ioc_macros$take_BANG_$fn__13264.invoke(ioc_macros.clj:953)
	at clojure.core.async.impl.channels.ManyToManyChannel$fn__8952.invoke(channels.clj:133)
	at clojure.lang.AFn.run(AFn.java:24)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:744)
Caused by: org.jclouds.http.HttpResponseException: request: HEAD https://phalliday-opscenter-backups-4.s3.amazonaws.com/snapshots/728139a7-093b-40f7-96a9-767a178fa5a5/sstables/OpsCenter-pdps-jb-114-Statistics.db
HTTP/1.1 failed with response: HTTP/1.1 403 Forbidden
	at org.jclouds.aws.handlers.ParseAWSErrorFromXmlContent.handleError(ParseAWSErrorFromXmlContent.java:63)
	... 25 more
DEBUG [clojure-agent-send-off-pool-9] 2014-12-05 16:29:15,793 Sending message '["172.31.10.206",{"details":{"error":"HEAD
https:\/\/phalliday-opscenter-backups-4.s3.amazonaws.com\/snapshots\/728139a7-093b-40f7-96a9-767a178fa5a5\/sstables\/OpsCenter-pdps-jb-114-Statistics.db
HTTP\/1.1 -> HTTP\/1.1 403 Forbidden","filename":"OpsCenter-pdps-jb-114-Statistics.db","keyspace":"OpsCenter","cf":"pdps"},"state":"running","type":"sstable","destination_id":"380a2db2ac2042c5bd43043d8ae53b6f","parent_req":"eb335075-765a-46b0-bcb8-89188d93cb1d"}]'
to channel '/snapshot/status'.
ERROR [async-dispatch-31] 2014-12-05 16:29:15,888 Got exception HEAD https://phalliday-opscenter-backups-4.s3.amazonaws.com/snapshots/728139a7-093b-40f7-96a9-767a178fa5a5/sstables/OpsCenter-pdps-jb-114-Data.db
HTTP/1.1 -> HTTP/1.1 403 Forbidden while sending opsagent.backups.entities.SSTable@8ca4473b
to :380a2db2ac2042c5bd43043d8ae53b6f
org.jclouds.rest.AuthorizationException: HEAD https://phalliday-opscenter-backups-4.s3.amazonaws.com/snapshots/728139a7-093b-40f7-96a9-767a178fa5a5/sstables/OpsCenter-pdps-jb-114-Data.db
HTTP/1.1 -> HTTP/1.1 403 Forbidden
	at org.jclouds.aws.handlers.ParseAWSErrorFromXmlContent.refineException(ParseAWSErrorFromXmlContent.java:122)
	at org.jclouds.s3.handlers.ParseS3ErrorFromXmlContent.refineException(ParseS3ErrorFromXmlContent.java:94)
	at org.jclouds.aws.handlers.ParseAWSErrorFromXmlContent.handleError(ParseAWSErrorFromXmlContent.java:89)
	at org.jclouds.http.handlers.DelegatingErrorHandler.handleError(DelegatingErrorHandler.java:65)
	at org.jclouds.http.internal.BaseHttpCommandExecutorService.shouldContinue(BaseHttpCommandExecutorService.java:135)
	at org.jclouds.http.internal.BaseHttpCommandExecutorService.invoke(BaseHttpCommandExecutorService.java:105)
	at org.jclouds.rest.internal.InvokeSyncToAsyncHttpMethod.invoke(InvokeSyncToAsyncHttpMethod.java:128)
	at org.jclouds.rest.internal.InvokeSyncToAsyncHttpMethod.apply(InvokeSyncToAsyncHttpMethod.java:94)
	at org.jclouds.rest.internal.InvokeSyncToAsyncHttpMethod.apply(InvokeSyncToAsyncHttpMethod.java:55)
	at org.jclouds.rest.internal.DelegatesToInvocationFunction.handle(DelegatesToInvocationFunction.java:156)
	at org.jclouds.rest.internal.DelegatesToInvocationFunction.invoke(DelegatesToInvocationFunction.java:123)
	at com.sun.proxy.$Proxy48.objectExists(Unknown Source)
	at org.jclouds.s3.blobstore.S3BlobStore.blobExists(S3BlobStore.java:173)
	at org.jclouds.blobstore2$blob_exists_QMARK_.invoke(blobstore2.clj:238)
	at opsagent.backups.destinations$create_blob.invoke(destinations.clj:43)
	at opsagent.backups.destinations$eval15431$fn__15460.invoke(destinations.clj:123)
	at opsagent.backups.destinations$eval15244$fn__15245$G__15235__15254.invoke(destinations.clj:19)
	at opsagent.backups.staging$start_staging_BANG_$fn__15649$state_machine__13248__auto____15650$fn__15655$fn__15683.invoke(staging.clj:46)
	at opsagent.backups.staging$start_staging_BANG_$fn__15649$state_machine__13248__auto____15650$fn__15655.invoke(staging.clj:44)
	at opsagent.backups.staging$start_staging_BANG_$fn__15649$state_machine__13248__auto____15650.invoke(staging.clj:40)
	at clojure.core.async.impl.ioc_macros$run_state_machine.invoke(ioc_macros.clj:940)
	at clojure.core.async.impl.ioc_macros$run_state_machine_wrapped.invoke(ioc_macros.clj:944)
	at clojure.core.async.impl.ioc_macros$take_BANG_$fn__13264.invoke(ioc_macros.clj:953)
	at clojure.core.async.impl.channels.ManyToManyChannel$fn__8952.invoke(channels.clj:133)
	at clojure.lang.AFn.run(AFn.java:24)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:744)
Caused by: org.jclouds.http.HttpResponseException: request: HEAD https://phalliday-opscenter-backups-4.s3.amazonaws.com/snapshots/728139a7-093b-40f7-96a9-767a178fa5a5/sstables/OpsCenter-pdps-jb-114-Data.db
HTTP/1.1 failed with response: HTTP/1.1 403 Forbidden
	at org.jclouds.aws.handlers.ParseAWSErrorFromXmlContent.handleError(ParseAWSErrorFromXmlContent.java:63)
	... 25 more
DEBUG [clojure-agent-send-off-pool-9] 2014-12-05 16:29:15,890 Sending message '["172.31.10.206",{"details":{"error":"HEAD
https:\/\/phalliday-opscenter-backups-4.s3.amazonaws.com\/snapshots\/728139a7-093b-40f7-96a9-767a178fa5a5\/sstables\/OpsCenter-pdps-jb-114-Data.db
HTTP\/1.1 -> HTTP\/1.1 403 Forbidden","filename":"OpsCenter-pdps-jb-114-Data.db","keyspace":"OpsCenter","cf":"pdps"},"state":"running","type":"sstable","destination_id":"380a2db2ac2042c5bd43043d8ae53b6f","parent_req":"eb335075-765a-46b0-bcb8-89188d93cb1d"}]'
to channel '/snapshot/status'.
DEBUG [node-details-1] 2014-12-05 16:29:16,030 Starting real-time collection
DEBUG [node-details-1] 2014-12-05 16:29:16,030 Collecting real-time data
DEBUG [clojure-agent-send-off-pool-9] 2014-12-05 16:29:16,034 Sending message '["172.31.10.206",{"os-load":0.07}]'
to channel '/node-details'.
DEBUG [clojure-agent-send-off-pool-9] 2014-12-05 16:29:16,298 Sending message '["172.31.10.206",{"destination_id":"380a2db2ac2042c5bd43043d8ae53b6f","type":"schema","state":"running","details":{"keyspace":"OpsCenter","filename":"OpsCenter\/schema.json"},"parent_req":"eb335075-765a-46b0-bcb8-89188d93cb1d"}]'
to channel '/snapshot/status'.
DEBUG [async-dispatch-31] 2014-12-05 16:29:16,402 Uploading file: snapshots/728139a7-093b-40f7-96a9-767a178fa5a5/sstables/OpsCenter-pdps-jb-115-Summary.db
DEBUG [async-dispatch-31] 2014-12-05 16:29:16,404 Throttle value requested is out of range,
using default.
ERROR [async-dispatch-31] 2014-12-05 16:29:16,510 Dec 05, 2014 4:29:16 PM org.jclouds.logging.jdk.JDKLogger
logError
SEVERE: Cannot retry after server error, command is not replayable: [method=org.jclouds.aws.s3.AWSS3AsyncClient.public
abstract com.google.common.util.concurrent.ListenableFuture org.jclouds.s3.S3AsyncClient.putObject(java.lang.String,org.jclouds.s3.domain.S3Object,org.jclouds.s3.options.PutObjectOptions[])[phalliday-opscenter-backups-4,
[metadata=[key=snapshots/728139a7-093b-40f7-96a9-767a178fa5a5/sstables/OpsCenter-pdps-jb-115-Summary.db,
bucket=null, uri=null, eTag=null, cacheControl=null, contentMetadata=[contentDisposition=null,
contentEncoding=null, contentLanguage=null, contentLength=107, contentMD5=null, contentType=application/octet-stream,
expires=null], lastModified=null, owner=null, storageClass=STANDARD, userMetadata={}]], [Lorg.jclouds.s3.options.PutObjectOptions;@4e0ecbfe],
request=PUT https://phalliday-opscenter-backups-4.s3.amazonaws.com/snapshots/728139a7-093b-40f7-96a9-767a178fa5a5/sstables/OpsCenter-pdps-jb-115-Summary.db
HTTP/1.1]

DEBUG [node-details-1] 2014-12-05 16:29:21,034 Starting real-time collection
DEBUG [node-details-1] 2014-12-05 16:29:21,035 Collecting real-time data
DEBUG [node-details-1] 2014-12-05 16:29:21,038 Send requested, but nothing new to send





 <http://www.datastax.com/>
PETER HALLIDAY
OpsCenter | Sr Software Engineer | 607.329.6905 | peter@datastax.com

 <https://twitter.com/phalliday>  <https://www.linkedin.com/profile/view?id=7030852>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message