cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "C. Scott Andreas (JIRA)" <j...@apache.org>
Subject [jira] [Resolved] (CASSANDRA-11982) Cassandra 3.5 cluster join fails
Date Mon, 19 Nov 2018 02:26:00 GMT

     [ https://issues.apache.org/jira/browse/CASSANDRA-11982?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

C. Scott Andreas resolved CASSANDRA-11982.
------------------------------------------
    Resolution: Information Provided

> Cassandra 3.5 cluster join fails
> --------------------------------
>
>                 Key: CASSANDRA-11982
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-11982
>             Project: Cassandra
>          Issue Type: Bug
>         Environment: # java -version
> openjdk version "1.8.0_91"
> OpenJDK Runtime Environment (build 1.8.0_91-b14)
> OpenJDK 64-Bit Server VM (build 25.91-b14, mixed mode)
> # uname -a
> Linux cassandra-dfs-10-10-160-19 4.4.11-23.53.amzn1.x86_64 #1 SMP Wed Jun 1 22:22:50
UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
> # rpm -qa datastax-ddc
> datastax-ddc-3.5.0-1.noarch
>            Reporter: Victor Trac
>            Priority: Major
>
> In an effort to work around a failing upgradesstables/scrub (https://issues.apache.org/jira/browse/CASSANDRA-11981),
I force-removed a node from the cluster, deleted its data directory, and tried to join a node
to the cluster anew. After a few hours, the bootstrap thread failed:
> cassandra.log
> {code}
> INFO  18:56:24 Redistributing index summaries
> INFO  19:56:23 Saved KeyCache (17 items) in 386 ms
> INFO  19:56:24 Redistributing index summaries
> ERROR 20:40:27 [Stream #b1e4d290-2d91-11e6-8904-df8dbad02c2a] Remote peer 10.10.160.18
failed stream session.
> INFO  20:40:27 [Stream #b1e4d290-2d91-11e6-8904-df8dbad02c2a] Session with /10.10.160.18
is complete
> WARN  20:40:27 [Stream #b1e4d290-2d91-11e6-8904-df8dbad02c2a] Stream failed
> ERROR 20:40:27 Error while waiting on bootstrap to complete. Bootstrap will have to be
restarted.
> java.util.concurrent.ExecutionException: org.apache.cassandra.streaming.StreamException:
Stream failed
>         at com.google.common.util.concurrent.AbstractFuture$Sync.getValue(AbstractFuture.java:299)
~[guava-18.0.jar:na]
>         at com.google.common.util.concurrent.AbstractFuture$Sync.get(AbstractFuture.java:286)
~[guava-18.0.jar:na]
>         at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:116)
~[guava-18.0.jar:na]
>         at org.apache.cassandra.service.StorageService.bootstrap(StorageService.java:1387)
[apache-cassandra-3.5.0.jar:3.5.0]
>         at org.apache.cassandra.service.StorageService.joinTokenRing(StorageService.java:974)
[apache-cassandra-3.5.0.jar:3.5.0]
>         at org.apache.cassandra.service.StorageService.initServer(StorageService.java:748)
[apache-cassandra-3.5.0.jar:3.5.0]
>         at org.apache.cassandra.service.StorageService.initServer(StorageService.java:613)
[apache-cassandra-3.5.0.jar:3.5.0]
>         at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:349)
[apache-cassandra-3.5.0.jar:3.5.0]
>         at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:551)
[apache-cassandra-3.5.0.jar:3.5.0]
>         at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:680)
[apache-cassandra-3.5.0.jar:3.5.0]
> Caused by: org.apache.cassandra.streaming.StreamException: Stream failed
>         at org.apache.cassandra.streaming.management.StreamEventJMXNotifier.onFailure(StreamEventJMXNotifier.java:85)
~[apache-cassandra-3.5.0.jar:3.5.0]
>         at com.google.common.util.concurrent.Futures$6.run(Futures.java:1310) ~[guava-18.0.jar:na]
>         at com.google.common.util.concurrent.MoreExecutors$DirectExecutor.execute(MoreExecutors.java:457)
~[guava-18.0.jar:na]
>         at com.google.common.util.concurrent.ExecutionList.executeListener(ExecutionList.java:156)
~[guava-18.0.jar:na]
>         at com.google.common.util.concurrent.ExecutionList.execute(ExecutionList.java:145)
~[guava-18.0.jar:na]
>         at com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:202)
~[guava-18.0.jar:na]
>         at org.apache.cassandra.streaming.StreamResultFuture.maybeComplete(StreamResultFuture.java:213)
~[apache-cassandra-3.5.0.jar:3.5.0]
>         at org.apache.cassandra.streaming.StreamResultFuture.handleSessionComplete(StreamResultFuture.java:189)
~[apache-cassandra-3.5.0.jar:3.5.0]
>         at org.apache.cassandra.streaming.StreamSession.closeSession(StreamSession.java:429)
~[apache-cassandra-3.5.0.jar:3.5.0]
>         at org.apache.cassandra.streaming.StreamSession.sessionFailed(StreamSession.java:636)
~[apache-cassandra-3.5.0.jar:3.5.0]
>         at org.apache.cassandra.streaming.StreamSession.messageReceived(StreamSession.java:489)
~[apache-cassandra-3.5.0.jar:3.5.0]
>         at org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:274)
~[apache-cassandra-3.5.0.jar:3.5.0]
>         at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_91]
> Jun 08, 2016 8:40:27 PM com.google.common.util.concurrent.ExecutionList executeListener
> SEVERE: RuntimeException while executing runnable com.google.common.util.concurrent.Futures$6@3592d1b5
with executor INSTANCE
> java.lang.NullPointerException
>         at org.apache.cassandra.service.StorageService$2.onFailure(StorageService.java:1382)
>         at com.google.common.util.concurrent.Futures$6.run(Futures.java:1310)
>         at com.google.common.util.concurrent.MoreExecutors$DirectExecutor.execute(MoreExecutors.java:457)
>         at com.google.common.util.concurrent.ExecutionList.executeListener(ExecutionList.java:156)
>         at com.google.common.util.concurrent.ExecutionList.execute(ExecutionList.java:145)
>         at com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:202)
>         at org.apache.cassandra.streaming.StreamResultFuture.maybeComplete(StreamResultFuture.java:213)
>         at org.apache.cassandra.streaming.StreamResultFuture.handleSessionComplete(StreamResultFuture.java:189)
>         at org.apache.cassandra.streaming.StreamSession.closeSession(StreamSession.java:429)
>         at org.apache.cassandra.streaming.StreamSession.sessionFailed(StreamSession.java:636)
>         at org.apache.cassandra.streaming.StreamSession.messageReceived(StreamSession.java:489)
>         at org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:274)
>         at java.lang.Thread.run(Thread.java:745)
> WARN  20:40:27 Some data streaming failed. Use nodetool to check bootstrap state and
resume. For more, see `nodetool help bootstrap`. IN_PROGRESS
> INFO  20:40:27 Waiting for gossip to settle before accepting client requests...
> INFO  20:40:35 No gossip backlog; proceeding
> {code}
> debug.log
> {code}
> DEBUG [GossipStage:1] 2016-06-08 20:40:27,370 FailureDetector.java:456 - Ignoring interval
time of 2001208211 for /10.240.0.4
> DEBUG [STREAM-IN-/10.10.160.18] 2016-06-08 20:40:27,701 ConnectionHandler.java:269 -
[Stream #b1e4d290-2d91-11e6-8904-df8dbad02c2a] Received Session Failed
> ERROR [STREAM-IN-/10.10.160.18] 2016-06-08 20:40:27,701 StreamSession.java:635 - [Stream
#b1e4d290-2d91-11e6-8904-df8dbad02c2a] Remote peer 10.10.160.18 failed stream session.
> DEBUG [STREAM-IN-/10.10.160.18] 2016-06-08 20:40:27,708 ConnectionHandler.java:110 -
[Stream #b1e4d290-2d91-11e6-8904-df8dbad02c2a] Closing stream connection handler on /10.10.160.18
> INFO  [STREAM-IN-/10.10.160.18] 2016-06-08 20:40:27,708 StreamResultFuture.java:185 -
[Stream #b1e4d290-2d91-11e6-8904-df8dbad02c2a] Session with /10.10.160.18 is complete
> DEBUG [STREAM-IN-/10.10.160.18] 2016-06-08 20:40:27,709 StreamCoordinator.java:146 -
Finished connecting all sessions
> WARN  [STREAM-IN-/10.10.160.18] 2016-06-08 20:40:27,710 StreamResultFuture.java:212 -
[Stream #b1e4d290-2d91-11e6-8904-df8dbad02c2a] Stream failed
> ERROR [main] 2016-06-08 20:40:27,711 StorageService.java:1392 - Error while waiting on
bootstrap to complete. Bootstrap will have to be restarted.
> java.util.concurrent.ExecutionException: org.apache.cassandra.streaming.StreamException:
Stream failed
>         at com.google.common.util.concurrent.AbstractFuture$Sync.getValue(AbstractFuture.java:299)
~[guava-18.0.jar:na]
>         at com.google.common.util.concurrent.AbstractFuture$Sync.get(AbstractFuture.java:286)
~[guava-18.0.jar:na]
>         at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:116)
~[guava-18.0.jar:na]
>         at org.apache.cassandra.service.StorageService.bootstrap(StorageService.java:1387)
[apache-cassandra-3.5.0.jar:3.5.0]
>         at org.apache.cassandra.service.StorageService.joinTokenRing(StorageService.java:974)
[apache-cassandra-3.5.0.jar:3.5.0]
>         at org.apache.cassandra.service.StorageService.initServer(StorageService.java:748)
[apache-cassandra-3.5.0.jar:3.5.0]
>         at org.apache.cassandra.service.StorageService.initServer(StorageService.java:613)
[apache-cassandra-3.5.0.jar:3.5.0]
>         at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:349)
[apache-cassandra-3.5.0.jar:3.5.0]
>         at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:551)
[apache-cassandra-3.5.0.jar:3.5.0]
>         at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:680)
[apache-cassandra-3.5.0.jar:3.5.0]
> Caused by: org.apache.cassandra.streaming.StreamException: Stream failed
>         at org.apache.cassandra.streaming.management.StreamEventJMXNotifier.onFailure(StreamEventJMXNotifier.java:85)
~[apache-cassandra-3.5.0.jar:3.5.0]
>         at com.google.common.util.concurrent.Futures$6.run(Futures.java:1310) ~[guava-18.0.jar:na]
>         at com.google.common.util.concurrent.MoreExecutors$DirectExecutor.execute(MoreExecutors.java:457)
~[guava-18.0.jar:na]
>         at com.google.common.util.concurrent.ExecutionList.executeListener(ExecutionList.java:156)
~[guava-18.0.jar:na]
>         at com.google.common.util.concurrent.ExecutionList.execute(ExecutionList.java:145)
~[guava-18.0.jar:na]
>         at com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:202)
~[guava-18.0.jar:na]
>         at org.apache.cassandra.streaming.StreamResultFuture.maybeComplete(StreamResultFuture.java:213)
~[apache-cassandra-3.5.0.jar:3.5.0]
>         at org.apache.cassandra.streaming.StreamResultFuture.handleSessionComplete(StreamResultFuture.java:189)
~[apache-cassandra-3.5.0.jar:3.5.0]
>         at org.apache.cassandra.streaming.StreamSession.closeSession(StreamSession.java:429)
~[apache-cassandra-3.5.0.jar:3.5.0]
>         at org.apache.cassandra.streaming.StreamSession.sessionFailed(StreamSession.java:636)
~[apache-cassandra-3.5.0.jar:3.5.0]
>         at org.apache.cassandra.streaming.StreamSession.messageReceived(StreamSession.java:489)
~[apache-cassandra-3.5.0.jar:3.5.0]
>         at org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:274)
~[apache-cassandra-3.5.0.jar:3.5.0]
>         at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_91]
> WARN  [main] 2016-06-08 20:40:27,777 StorageService.java:1029 - Some data streaming failed.
Use nodetool to check bootstrap state and resume. For more, see `nodetool help bootstrap`.
IN_PROGRESS
> INFO  [main] 2016-06-08 20:40:27,779 CassandraDaemon.java:639 - Waiting for gossip to
settle before accepting client requests...
> DEBUG [GossipStage:1] 2016-06-08 20:40:31,371 FailureDetector.java:456 - Ignoring interval
time of 2256483778 for /10.10.128.18
> DEBUG [GossipStage:1] 2016-06-08 20:40:31,371 FailureDetector.java:456 - Ignoring interval
time of 2001387252 for /10.10.160.18
> {code} 
> system.log
> {code}
> INFO  [IndexSummaryManager:1] 2016-06-08 19:56:24,495 IndexSummaryRedistribution.java:74
- Redistributing index summaries
> ERROR [STREAM-IN-/10.10.160.18] 2016-06-08 20:40:27,701 StreamSession.java:635 - [Stream
#b1e4d290-2d91-11e6-8904-df8dbad02c2a] Remote peer 10.10.160.18 failed stream session.
> INFO  [STREAM-IN-/10.10.160.18] 2016-06-08 20:40:27,708 StreamResultFuture.java:185 -
[Stream #b1e4d290-2d91-11e6-8904-df8dbad02c2a] Session with /10.10.160.18 is complete
> WARN  [STREAM-IN-/10.10.160.18] 2016-06-08 20:40:27,710 StreamResultFuture.java:212 -
[Stream #b1e4d290-2d91-11e6-8904-df8dbad02c2a] Stream failed
> ERROR [main] 2016-06-08 20:40:27,711 StorageService.java:1392 - Error while waiting on
bootstrap to complete. Bootstrap will have to be restarted.
> java.util.concurrent.ExecutionException: org.apache.cassandra.streaming.StreamException:
Stream failed
>         at com.google.common.util.concurrent.AbstractFuture$Sync.getValue(AbstractFuture.java:299)
~[guava-18.0.jar:na]
>         at com.google.common.util.concurrent.AbstractFuture$Sync.get(AbstractFuture.java:286)
~[guava-18.0.jar:na]
>         at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:116)
~[guava-18.0.jar:na]
>         at org.apache.cassandra.service.StorageService.bootstrap(StorageService.java:1387)
[apache-cassandra-3.5.0.jar:3.5.0]
>         at org.apache.cassandra.service.StorageService.joinTokenRing(StorageService.java:974)
[apache-cassandra-3.5.0.jar:3.5.0]
>         at org.apache.cassandra.service.StorageService.initServer(StorageService.java:748)
[apache-cassandra-3.5.0.jar:3.5.0]
>         at org.apache.cassandra.service.StorageService.initServer(StorageService.java:613)
[apache-cassandra-3.5.0.jar:3.5.0]
>         at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:349)
[apache-cassandra-3.5.0.jar:3.5.0]
>         at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:551)
[apache-cassandra-3.5.0.jar:3.5.0]
>         at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:680)
[apache-cassandra-3.5.0.jar:3.5.0]
> Caused by: org.apache.cassandra.streaming.StreamException: Stream failed
>         at org.apache.cassandra.streaming.management.StreamEventJMXNotifier.onFailure(StreamEventJMXNotifier.java:85)
~[apache-cassandra-3.5.0.jar:3.5.0]
>         at com.google.common.util.concurrent.Futures$6.run(Futures.java:1310) ~[guava-18.0.jar:na]
>         at com.google.common.util.concurrent.MoreExecutors$DirectExecutor.execute(MoreExecutors.java:457)
~[guava-18.0.jar:na]
>         at com.google.common.util.concurrent.ExecutionList.executeListener(ExecutionList.java:156)
~[guava-18.0.jar:na]
>         at com.google.common.util.concurrent.ExecutionList.execute(ExecutionList.java:145)
~[guava-18.0.jar:na]
>         at com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:202)
~[guava-18.0.jar:na]
>         at org.apache.cassandra.streaming.StreamResultFuture.maybeComplete(StreamResultFuture.java:213)
~[apache-cassandra-3.5.0.jar:3.5.0]
>         at org.apache.cassandra.streaming.StreamResultFuture.handleSessionComplete(StreamResultFuture.java:189)
~[apache-cassandra-3.5.0.jar:3.5.0]
>         at org.apache.cassandra.streaming.StreamSession.closeSession(StreamSession.java:429)
~[apache-cassandra-3.5.0.jar:3.5.0]
>         at org.apache.cassandra.streaming.StreamSession.sessionFailed(StreamSession.java:636)
~[apache-cassandra-3.5.0.jar:3.5.0]
>         at org.apache.cassandra.streaming.StreamSession.messageReceived(StreamSession.java:489)
~[apache-cassandra-3.5.0.jar:3.5.0]
>         at org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:274)
~[apache-cassandra-3.5.0.jar:3.5.0]
>         at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_91]
> WARN  [main] 2016-06-08 20:40:27,777 StorageService.java:1029 - Some data streaming failed.
Use nodetool to check bootstrap state and resume. For more, see `nodetool help bootstrap`.
IN_PROGRESS
> INFO  [main] 2016-06-08 20:40:27,779 CassandraDaemon.java:639 - Waiting for gossip to
settle before accepting client requests...
> INFO  [main] 2016-06-08 20:40:35,780 CassandraDaemon.java:670 - No gossip backlog; proceeding
> INFO  [main] 2016-06-08 20:40:35,876 NativeTransportService.java:70 - Netty using native
Epoll event loop
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org
For additional commands, e-mail: commits-help@cassandra.apache.org


Mime
View raw message