spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From James King <jakwebin...@gmail.com>
Subject Re: Worker Spark Port
Date Fri, 15 May 2015 13:30:44 GMT
I think this answers my question

"executors, on the other hand, are bound with an application, ie spark
context. Thus you modify executor properties through a context."

Many Thanks.

jk

On Fri, May 15, 2015 at 3:23 PM, ayan guha <guha.ayan@gmail.com> wrote:

> Hi
>
> I think you are mixing things a bit.
>
> Worker is part of the cluster. So it is governed by cluster manager. If
> you are running standalone cluster, then you can modify spark-env and
> configure SPARK_WORKER_PORT.
>
> executors, on the other hand, are bound with an application, ie spark
> context. Thus you modify executor properties through a context.
>
> So, master != driver and executor != worker.
>
> Best
> Ayan
>
> On Fri, May 15, 2015 at 7:52 PM, James King <jakwebinbox@gmail.com> wrote:
>
>> So I'm using code like this to use specific ports:
>>
>> val conf = new SparkConf()
>>     .setMaster(master)
>>     .setAppName("namexxx")
>>     .set("spark.driver.port", "51810")
>>     .set("spark.fileserver.port", "51811")
>>     .set("spark.broadcast.port", "51812")
>>     .set("spark.replClassServer.port", "51813")
>>     .set("spark.blockManager.port", "51814")
>>     .set("spark.executor.port", "51815")
>>
>> My question now is : Will the master forward the spark.executor.port value (to use)
to the worker when it hands it a task to do?
>>
>> Also the property spark.executor.port is different from the Worker spark port, how
can I make the Worker run on a specific port?
>>
>> Regards
>>
>> jk
>>
>>
>> On Wed, May 13, 2015 at 7:51 PM, James King <jakwebinbox@gmail.com>
>> wrote:
>>
>>> Indeed, many thanks.
>>>
>>>
>>> On Wednesday, 13 May 2015, Cody Koeninger <cody@koeninger.org> wrote:
>>>
>>>> I believe most ports are configurable at this point, look at
>>>>
>>>> http://spark.apache.org/docs/latest/configuration.html
>>>>
>>>> search for ".port"
>>>>
>>>> On Wed, May 13, 2015 at 9:38 AM, James King <jakwebinbox@gmail.com>
>>>> wrote:
>>>>
>>>>> I understated that this port value is randomly selected.
>>>>>
>>>>> Is there a way to enforce which spark port a Worker should use?
>>>>>
>>>>
>>>>
>>
>
>
> --
> Best Regards,
> Ayan Guha
>

Mime
View raw message