spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Aaron Babcock <aaron.babc...@gmail.com>
Subject spark through vpn, SPARK_LOCAL_IP
Date Sat, 05 Oct 2013 21:45:31 GMT
Hello,

I am using spark through a vpn. My driver machine ends up with two ip
addresses, one routable from the cluster and one not.

Things generally work when I set the SPARK_LOCAL_IP environment
variable to the proper ip address.

However, when I try to use the take function ie: myRdd.take(1),  I run
into a hiccup. From the logfiles on the workers I can see that they
trying to connect to the nonroutable ip address, they are not
respecting SPARK_LOCAL_IP somehow.

Here is the relevant worker log snippet, 192.168.250.47 is the correct
routable ip address of the driver, 192.168.0.7 is the incorrect
address of the driver. Any thoughts about what else I need to
configure?

13/10/05 16:17:36 INFO ConnectionManager: Accepted connection from
[192.168.250.47/192.168.250.47]
13/10/05 16:18:41 WARN SendingConnection: Error finishing connection
to /192.168.0.7:60513
java.net.ConnectException: Connection timed out
at sun.nio.ch.SocketChannelImpl.$$YJP$$checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.checkConnect(SocketChannelImpl.java)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:567)
at spark.network.SendingConnection.finishConnect(Connection.scala:221)
at spark.network.ConnectionManager.spark$network$ConnectionManager$$run(ConnectionManager.scala:127)
at spark.network.ConnectionManager$$anon$4.run(ConnectionManager.scala:70)
13/10/05 16:18:41 INFO ConnectionManager: Handling connection error on
connection to ConnectionManagerId(192.168.0.7,60513)
13/10/05 16:18:41 INFO ConnectionManager: Removing SendingConnection
to ConnectionManagerId(192.168.0.7,60513)

Mime
View raw message