cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ariel Weisberg (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (CASSANDRA-14522) sstableloader should use discovered broadcast address to connect to intracluster
Date Thu, 02 Aug 2018 16:38:00 GMT

    [ https://issues.apache.org/jira/browse/CASSANDRA-14522?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16567046#comment-16567046
] 

Ariel Weisberg commented on CASSANDRA-14522:
--------------------------------------------

In 2.2 the Java driver doesn't provide the information from {{system.peers}} I think. It only
provides {{getAddress()}}. So it will need to go and query {{system.peers}}.

[~Yarnspinner] in your patch you read {{local}} and {{peers}} but with the driver's load balancing
it's not guaranteed that {{local}} and {{peers}} will be read from the same node so you can
still miss one. I am not sure if the {{listenAddress}} value is better? It's not broadcast
address so it's not 100% correct.

It's kind of a mess because the comments in that version of the driver say this:
{code:java}
    // The listen_address (really, the broadcast one) as know by Cassandra. We use that internally
because
    // that's the 'peer' in the 'System.peers' table and avoids querying the full peers table
in
    // ControlConnection.refreshNodeInfo. We don't want to expose however because we don't
always have the info
    // (partly because the 'System.local' doesn't have it for some weird reason for instance).
    volatile InetAddress listenAddress;
{code}

Not quite sure how to fix that in 2.2. Maybe you can research how to fix this in 2.2.

> sstableloader should use discovered broadcast address to connect to intracluster
> --------------------------------------------------------------------------------
>
>                 Key: CASSANDRA-14522
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-14522
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Tools
>            Reporter: Jeremy Hanna
>            Assignee: Jeremy
>            Priority: Major
>              Labels: lhf
>         Attachments: CASSANDRA-14522.patch
>
>
> Currently, in the LoaderOptions for the BulkLoader, the user can give a list of initial
host addresses.  That's to do the initial connection to the cluster but also to stream the
sstables.  If you have two physical interfaces, one for rpc, the other for internode traffic,
then bulk loader won't currently work.  It will throw an error such as:
> {quote}
> > sstableloader -v -u cassadmin -pw xxx -d 10.133.210.101,10.133.210.102,10.133.210.103,10.133.210.104
/var/lib/cassandra/commitlog/backup_tmp/test_bkup/bkup_tbl
> Established connection to initial hosts
> Opening sstables and calculating sections to stream
> Streaming relevant part of /var/lib/cassandra/commitlog/backup_tmp/test_bkup/bkup_tbl/mc-1-big-Data.db
/var/lib/cassandra/commitlog/backup_tmp/test_bkup/bkup_tbl/mc-2-big-Data.db  to [/10.133.210.101,
/10.133.210.103, /10.133.210.102, /10.133.210.104]
> progress: total: 100% 0  MB/s(avg: 0 MB/s)ERROR 10:16:05,311 [Stream #9ed00130-6ff6-11e8-965c-93a78bf96e60]
Streaming error occurred
> java.net.ConnectException: Connection refused
>         at sun.nio.ch.Net.connect0(Native Method) ~[na:1.8.0_101]
>         at sun.nio.ch.Net.connect(Net.java:454) ~[na:1.8.0_101]
>         at sun.nio.ch.Net.connect(Net.java:446) ~[na:1.8.0_101]
>         at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:648) ~[na:1.8.0_101]
>         at java.nio.channels.SocketChannel.open(SocketChannel.java:189) ~[na:1.8.0_101]
>         at org.apache.cassandra.tools.BulkLoadConnectionFactory.createConnection(BulkLoadConnectionFactory.java:60)
~[cassandra-all-3.0.15.2128.jar:3.0.15.2128]
>         at org.apache.cassandra.streaming.StreamSession.createConnection(StreamSession.java:266)
~[cassandra-all-3.0.15.2128.jar:3.0.15.2128]
>         at org.apache.cassandra.streaming.ConnectionHandler.initiate(ConnectionHandler.java:86)
~[cassandra-all-3.0.15.2128.jar:3.0.15.2128]
>         at org.apache.cassandra.streaming.StreamSession.start(StreamSession.java:253)
~[cassandra-all-3.0.15.2128.jar:3.0.15.2128]
>         at org.apache.cassandra.streaming.StreamCoordinator$StreamSessionConnector.run(StreamCoordinator.java:212)
[cassandra-all-3.0.15.2128.jar:3.0.15.2128]
>         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
[na:1.8.0_101]
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[na:1.8.0_101]
>         at org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:79)
[cassandra-all-3.0.15.2128.jar:3.0.15.2128]
>         at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
~[netty-all-4.0.54.Final.jar:4.0.54.Final]
>         at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_101]
> ERROR 10:16:05,312 [Stream #9ed00130-6ff6-11e8-965c-93a78bf96e60] Streaming error occurred
> java.net.ConnectException: Connection refused
>         at sun.nio.ch.Net.connect0(Native Method) ~[na:1.8.0_101]
>         at sun.nio.ch.Net.connect(Net.java:454) ~[na:1.8.0_101]
>         at sun.nio.ch.Net.connect(Net.java:446) ~[na:1.8.0_101]
>         at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:648) ~[na:1.8.0_101]
>         at java.nio.channels.SocketChannel.open(SocketChannel.java:189) ~[na:1.8.0_101]
>         at org.apache.cassandra.tools.BulkLoadConnectionFactory.createConnection(BulkLoadConnectionFactory.java:60)
~[cassandra-all-3.0.15.2128.jar:3.0.15.2128]
>         at org.apache.cassandra.streaming.StreamSession.createConnection(StreamSession.java:266)
~[cassandra-all-3.0.15.2128.jar:3.0.15.2128]
>         at org.apache.cassandra.streaming.ConnectionHandler.initiate(ConnectionHandler.java:86)
~[cassandra-all-3.0.15.2128.jar:3.0.15.2128]
>         at org.apache.cassandra.streaming.StreamSession.start(StreamSession.java:253)
~[cassandra-all-3.0.15.2128.jar:3.0.15.2128]
>         at org.apache.cassandra.streaming.StreamCoordinator$StreamSessionConnector.run(StreamCoordinator.java:212)
[cassandra-all-3.0.15.2128.jar:3.0.15.2128]
>         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
[na:1.8.0_101]
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[na:1.8.0_101]
>         at org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:79)
[cassandra-all-3.0.15.2128.jar:3.0.15.2128]
>         at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
~[netty-all-4.0.54.Final.jar:4.0.54.Final]
>         at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_101]
> ERROR 10:16:05,312 [Stream #9ed00130-6ff6-11e8-965c-93a78bf96e60] Streaming error occurred
> java.net.ConnectException: Connection refused
>         at sun.nio.ch.Net.connect0(Native Method) ~[na:1.8.0_101]
>         at sun.nio.ch.Net.connect(Net.java:454) ~[na:1.8.0_101]
>         at sun.nio.ch.Net.connect(Net.java:446) ~[na:1.8.0_101]
>         at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:648) ~[na:1.8.0_101]
>         at java.nio.channels.SocketChannel.open(SocketChannel.java:189) ~[na:1.8.0_101]
>         at org.apache.cassandra.tools.BulkLoadConnectionFactory.createConnection(BulkLoadConnectionFactory.java:60)
~[cassandra-all-3.0.15.2128.jar:3.0.15.2128]
>         at org.apache.cassandra.streaming.StreamSession.createConnection(StreamSession.java:266)
~[cassandra-all-3.0.15.2128.jar:3.0.15.2128]
>         at org.apache.cassandra.streaming.ConnectionHandler.initiate(ConnectionHandler.java:86)
~[cassandra-all-3.0.15.2128.jar:3.0.15.2128]
>         at org.apache.cassandra.streaming.StreamSession.start(StreamSession.java:253)
~[cassandra-all-3.0.15.2128.jar:3.0.15.2128]
>         at org.apache.cassandra.streaming.StreamCoordinator$StreamSessionConnector.run(StreamCoordinator.java:212)
[cassandra-all-3.0.15.2128.jar:3.0.15.2128]
>         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
[na:1.8.0_101]
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[na:1.8.0_101]
>         at org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:79)
[cassandra-all-3.0.15.2128.jar:3.0.15.2128]
>         at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
~[netty-all-4.0.54.Final.jar:4.0.54.Final]
>         at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_101]
> progress: total: 100% 0  MB/s(avg: 0 MB/s)WARN  10:16:05,320 [Stream #9ed00130-6ff6-11e8-965c-93a78bf96e60]
Stream failed
> Streaming to the following hosts failed:
> [/10.133.210.101, /10.133.210.103, /10.133.210.102]
> java.util.concurrent.ExecutionException: org.apache.cassandra.streaming.StreamException:
Stream failed
>         at com.google.common.util.concurrent.AbstractFuture$Sync.getValue(AbstractFuture.java:299)
>         at com.google.common.util.concurrent.AbstractFuture$Sync.get(AbstractFuture.java:286)
>         at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:116)
>         at org.apache.cassandra.tools.BulkLoader.main(BulkLoader.java:122)
>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>         at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>         at java.lang.reflect.Method.invoke(Method.java:498)
>         at com.datastax.bdp.tools.ShellToolWrapper.main(ShellToolWrapper.java:34)
> Caused by: org.apache.cassandra.streaming.StreamException: Stream failed
>         at org.apache.cassandra.streaming.management.StreamEventJMXNotifier.onFailure(StreamEventJMXNotifier.java:85)
>         at com.google.common.util.concurrent.Futures$6.run(Futures.java:1310)
>         at com.google.common.util.concurrent.MoreExecutors$DirectExecutor.execute(MoreExecutors.java:457)
>         at com.google.common.util.concurrent.ExecutionList.executeListener(ExecutionList.java:156)
>         at com.google.common.util.concurrent.ExecutionList.execute(ExecutionList.java:145)
>         at com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:202)
>         at org.apache.cassandra.streaming.StreamResultFuture.maybeComplete(StreamResultFuture.java:215)
>         at org.apache.cassandra.streaming.StreamResultFuture.handleSessionComplete(StreamResultFuture.java:191)
>         at org.apache.cassandra.streaming.StreamSession.closeSession(StreamSession.java:449)
>         at org.apache.cassandra.streaming.StreamSession.onError(StreamSession.java:549)
>         at org.apache.cassandra.streaming.StreamSession.start(StreamSession.java:259)
>         at org.apache.cassandra.streaming.StreamCoordinator$StreamSessionConnector.run(StreamCoordinator.java:212)
>         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>         at org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:79)
>         at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
>         at java.lang.Thread.run(Thread.java:745)
> WARN  10:16:05,322 [Stream #9ed00130-6ff6-11e8-965c-93a78bf96e60] Stream failed
> WARN  10:16:05,322 [Stream #9ed00130-6ff6-11e8-965c-93a78bf96e60] Stream failed
> {quote}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org
For additional commands, e-mail: commits-help@cassandra.apache.org


Mime
View raw message