spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From durin <m...@simon-schaefer.net>
Subject Re: Spark Streaming fails - where is the problem?
Date Wed, 06 Aug 2014 18:59:10 GMT
Hi Andrew, 
for this test I only have one machine which provides the master and only worker. 
So all I'd need is communication to the Internet to access the twitter API. 
I've tried assigning a specific port to the driver and creating iptables rules for this port,
but that didn't work. 
Best regards, 
Simon 
On Aug 6, 2014 11:37 AM, &quot;Andrew Or-2 [via Apache Spark User List]&quot; &lt;ml-node+s1001560n11561h89@n3.nabble.com&gt;
wrote: 

	Hi Simon, The drivers and executors currently choose random ports to talk to each other,
so the Spark nodes will have to have full TCP access to each other. This is changed in a very
recent commit, where all of these random ports will become configurable:  https://github.com/apache/spark/commit/09f7e4587bbdf74207d2629e8c1314f93d865999
. This will be available in Spark 1.1, but for now you will have to open all ports among the
nodes in your cluster. 
-Andrew 2014-08-06 10:23 GMT-07:00 durin &lt; [hidden email] &gt;: 
&lt;blockquote style='border-left:2px solid #CCCCCC;padding:0 1em' class=&quot;gmail_quote&quot;
style=&quot;margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex&quot;&gt;Update:
I can get it to work by disabling iptables temporarily. I can, 
however, not figure out on which port I have to accept traffic. 4040 and any 
of the Master or Worker ports mentioned in the previous post don&#39;t work. 

Can it be one of the randomly assigned ones in the 30k to 60k range? Those 
appear to change every time, making it difficult to apply any sensible 
rules. 



-- 
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Spark-Streaming-fails-where-is-the-problem-tp11355p11556.html


Sent from the Apache Spark User List mailing list archive at Nabble.com. 

--------------------------------------------------------------------- 
To unsubscribe, e-mail: [hidden email] 
For additional commands, e-mail: [hidden email] 




	
	
	
	
	
	
	
		If you reply to this email, your message will be added to the discussion below: 
		http://apache-spark-user-list.1001560.n3.nabble.com/Spark-Streaming-fails-where-is-the-problem-tp11355p11561.html

	
	
		
		To unsubscribe from Spark Streaming fails - where is the problem?, click here . 
		NAML 
	



--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Spark-Streaming-fails-where-is-the-problem-tp11355p11566.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
Mime
View raw message