spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jacob Eisinger <jeis...@us.ibm.com>
Subject Re: Comprehensive Port Configuration reference?
Date Tue, 06 May 2014 13:09:49 GMT

Howdy Scott,

Please see the discussions about securing the Spark network [1] [2].

In a nut shell, Spark opens up a couple of well known ports.  And,then the
workers and the shell open up dynamic ports for each job.  These dynamic
ports make securing the Spark network difficult.

Jacob

[1]
http://apache-spark-user-list.1001560.n3.nabble.com/Securing-Spark-s-Network-td4832.html
[2]
http://apache-spark-user-list.1001560.n3.nabble.com/spark-shell-driver-interacting-with-Workers-in-YARN-mode-firewall-blocking-communication-td5237.html

Jacob D. Eisinger
IBM Emerging Technologies
jeising@us.ibm.com - (512) 286-6075



From:	Scott Clasen <scott.clasen@gmail.com>
To:	user@spark.incubator.apache.org
Date:	05/05/2014 11:39 AM
Subject:	Comprehensive Port Configuration reference?



Is there somewhere documented how one would go about configuring every open
port a spark application needs?

This seems like one of the main things that make running spark hard in
places like EC2 where you arent using the canned spark scripts.

Starting an app looks like you'll see ports open for

BlockManager
OutoutTracker
FileServer
WebUI
Local port to get callbacks from mesos master..

What else?

How do I configure all of these?



--
View this message in context:
http://apache-spark-user-list.1001560.n3.nabble.com/Comprehensive-Port-Configuration-reference-tp5384.html

Sent from the Apache Spark User List mailing list archive at Nabble.com.


Mime
View raw message