Hi Andrew,
for this test I only have one machine which provides the master and only worker.
So all I'd need is communication to the Internet to access the twitter API.
I've tried assigning a specific port to the driver and creating iptables rules for this port,
but that didn't work.
Best regards,
Simon
On Aug 6, 2014 11:37 AM, "Andrew Or-2 [via Apache Spark User List]" <ml-node+s1001560n11561h89@n3.nabble.com>
wrote:
Hi Simon, The drivers and executors currently choose random ports to talk to each other,
so the Spark nodes will have to have full TCP access to each other. This is changed in a very
recent commit, where all of these random ports will become configurable: https://github.com/apache/spark/commit/09f7e4587bbdf74207d2629e8c1314f93d865999
. This will be available in Spark 1.1, but for now you will have to open all ports among the
nodes in your cluster.
-Andrew 2014-08-06 10:23 GMT-07:00 durin < [hidden email] >:
<blockquote style='border-left:2px solid #CCCCCC;padding:0 1em' class="gmail_quote"
style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Update:
I can get it to work by disabling iptables temporarily. I can,
however, not figure out on which port I have to accept traffic. 4040 and any
of the Master or Worker ports mentioned in the previous post don't work.
Can it be one of the randomly assigned ones in the 30k to 60k range? Those
appear to change every time, making it difficult to apply any sensible
rules.
--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Spark-Streaming-fails-where-is-the-problem-tp11355p11556.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]
If you reply to this email, your message will be added to the discussion below:
http://apache-spark-user-list.1001560.n3.nabble.com/Spark-Streaming-fails-where-is-the-problem-tp11355p11561.html
To unsubscribe from Spark Streaming fails - where is the problem?, click here .
NAML
--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Spark-Streaming-fails-where-is-the-problem-tp11355p11566.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
|