spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Aaron Babcock <>
Subject spark through vpn, SPARK_LOCAL_IP
Date Sat, 05 Oct 2013 21:45:31 GMT

I am using spark through a vpn. My driver machine ends up with two ip
addresses, one routable from the cluster and one not.

Things generally work when I set the SPARK_LOCAL_IP environment
variable to the proper ip address.

However, when I try to use the take function ie: myRdd.take(1),  I run
into a hiccup. From the logfiles on the workers I can see that they
trying to connect to the nonroutable ip address, they are not
respecting SPARK_LOCAL_IP somehow.

Here is the relevant worker log snippet, is the correct
routable ip address of the driver, is the incorrect
address of the driver. Any thoughts about what else I need to

13/10/05 16:17:36 INFO ConnectionManager: Accepted connection from
13/10/05 16:18:41 WARN SendingConnection: Error finishing connection
to / Connection timed out
at$$YJP$$checkConnect(Native Method)
13/10/05 16:18:41 INFO ConnectionManager: Handling connection error on
connection to ConnectionManagerId(,60513)
13/10/05 16:18:41 INFO ConnectionManager: Removing SendingConnection
to ConnectionManagerId(,60513)

View raw message