spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Andrew Ash <and...@andrewash.com>
Subject Re: Ports required for running spark
Date Fri, 01 Aug 2014 06:46:27 GMT
Also Konstantin do you have a firewall between your Spark services?  If
that's what's causing these issues, then you may be interested in the
ability to configure every port a Spark service listens on --
https://issues.apache.org/jira/browse/SPARK-2157


On Thu, Jul 31, 2014 at 8:47 AM, Haiyang Fu <haiyangfu512@gmail.com> wrote:

> Hi Konstantin,
>
> Could you please post your first container's stderr log here which is
> always the AM log?As far as I know, ports except 8020,8080,8081,50070,50071
> are all random socket ports determined by each job. So 33007   maybe just
> a temporary port for data transferation. The deeper reason for 33007
> ConnectionRefused  maybe something else.would you please check If it is
> always 33007 when you submit your job again ?
>
>
> haiy
>
>
> On Thu, Jul 31, 2014 at 8:17 PM, Konstantin Kudryavtsev <
> kudryavtsev.konstantin@gmail.com> wrote:
>
>> Hi Haiyang,
>>
>> you are right, YARN takes over the resource management, bot I constantly
>> got Exception ConnectionRefused on mentioned port. So, I suppose some spark
>> internal communications are done via this port... but I don't know what
>> exactly and how can I change it...
>>
>> Thank you,
>> Konstantin Kudryavtsev
>>
>>
>> On Thu, Jul 31, 2014 at 2:53 PM, Haiyang Fu <haiyangfu512@gmail.com>
>> wrote:
>>
>>> Hi Konstantin,
>>> Would you please post some more details? Error info or exception from
>>> the log on what situation´╝čwhen you run spark job on yarn cluster mode ,yarn
>>> will take over all the resource management.
>>>
>>>
>>> On Thu, Jul 31, 2014 at 6:17 PM, Konstantin Kudryavtsev <
>>> kudryavtsev.konstantin@gmail.com> wrote:
>>>
>>>> Hi Larry,
>>>>
>>>> I'm afraid this is standalone mode, I'm interesting in YARN
>>>>
>>>> Also, I don't see port-in-trouble 33007   which i believe related to
>>>> Akka
>>>>
>>>> Thank you,
>>>> Konstantin Kudryavtsev
>>>>
>>>>
>>>> On Thu, Jul 31, 2014 at 1:11 PM, Larry Xiao <xiaodi@sjtu.edu.cn> wrote:
>>>>
>>>>>  Hi Konstantin,
>>>>>
>>>>> I think you can find it at
>>>>> https://spark.apache.org/docs/latest/spark-standalone.html#configuring-ports-for-network-security
>>>>> and you can specify port for master or worker at conf/spark-env.sh
>>>>>
>>>>> Larry
>>>>>
>>>>>
>>>>> On 7/31/14, 6:04 PM, Konstantin Kudryavtsev wrote:
>>>>>
>>>>> Hi there,
>>>>>
>>>>>  I'm trying to run Spark on YARN cluster and face with issued that
>>>>> some ports are closed, particularly port 33007 (I suppose it's used
>>>>> by Akka)
>>>>>
>>>>>  Could you please provide me with a list of all ports required for
>>>>> Spark?
>>>>> Also, is it possible to set up these ports?
>>>>>
>>>>> Thank you in advance,
>>>>> Konstantin Kudryavtsev
>>>>>
>>>>>
>>>>>
>>>>
>>>
>>
>

Mime
View raw message