spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Haiyang Fu <haiyangfu...@gmail.com>
Subject Re: Ports required for running spark
Date Thu, 31 Jul 2014 12:47:29 GMT
Hi Konstantin,

Could you please post your first container's stderr log here which is
always the AM log?As far as I know, ports except 8020,8080,8081,50070,50071
are all random socket ports determined by each job. So 33007   maybe just a
temporary port for data transferation. The deeper reason for 33007
ConnectionRefused  maybe something else.would you please check If it is
always 33007 when you submit your job again ?


haiy


On Thu, Jul 31, 2014 at 8:17 PM, Konstantin Kudryavtsev <
kudryavtsev.konstantin@gmail.com> wrote:

> Hi Haiyang,
>
> you are right, YARN takes over the resource management, bot I constantly
> got Exception ConnectionRefused on mentioned port. So, I suppose some spark
> internal communications are done via this port... but I don't know what
> exactly and how can I change it...
>
> Thank you,
> Konstantin Kudryavtsev
>
>
> On Thu, Jul 31, 2014 at 2:53 PM, Haiyang Fu <haiyangfu512@gmail.com>
> wrote:
>
>> Hi Konstantin,
>> Would you please post some more details? Error info or exception from the
>> log on what situation´╝čwhen you run spark job on yarn cluster mode ,yarn
>> will take over all the resource management.
>>
>>
>> On Thu, Jul 31, 2014 at 6:17 PM, Konstantin Kudryavtsev <
>> kudryavtsev.konstantin@gmail.com> wrote:
>>
>>> Hi Larry,
>>>
>>> I'm afraid this is standalone mode, I'm interesting in YARN
>>>
>>> Also, I don't see port-in-trouble 33007   which i believe related to
>>> Akka
>>>
>>> Thank you,
>>> Konstantin Kudryavtsev
>>>
>>>
>>> On Thu, Jul 31, 2014 at 1:11 PM, Larry Xiao <xiaodi@sjtu.edu.cn> wrote:
>>>
>>>>  Hi Konstantin,
>>>>
>>>> I think you can find it at
>>>> https://spark.apache.org/docs/latest/spark-standalone.html#configuring-ports-for-network-security
>>>> and you can specify port for master or worker at conf/spark-env.sh
>>>>
>>>> Larry
>>>>
>>>>
>>>> On 7/31/14, 6:04 PM, Konstantin Kudryavtsev wrote:
>>>>
>>>> Hi there,
>>>>
>>>>  I'm trying to run Spark on YARN cluster and face with issued that
>>>> some ports are closed, particularly port 33007 (I suppose it's used by
>>>> Akka)
>>>>
>>>>  Could you please provide me with a list of all ports required for
>>>> Spark?
>>>> Also, is it possible to set up these ports?
>>>>
>>>> Thank you in advance,
>>>> Konstantin Kudryavtsev
>>>>
>>>>
>>>>
>>>
>>
>

Mime
View raw message