spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Maximiliano Patricio Méndez <mmen...@despegar.com>
Subject Re: Problems initializing SparkUI
Date Mon, 01 Aug 2016 21:03:12 GMT
Hi, thanks again for the answer.

Looking a little bit closer into that, I found out that the DriverWrapper
process was not running in the hostname the log reported. It is runnning,
but in another host. Mistery.

If I manually go to the host that has the DriverWrapper running in it, on
port 4040, I can see the sparkUI without problems, but if I go through the
master > applicationUI it tries to send me to the wrong host (the one the
driver reports in its log).

The hostname the driver reports is the same from which I send the submit
request.

2016-08-01 17:27 GMT-03:00 Mich Talebzadeh <mich.talebzadeh@gmail.com>:

> Fine.
>
> In that case which process is your driver program (from jps output)?
>
> Thanks
>
> Dr Mich Talebzadeh
>
>
>
> LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
> <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
>
>
>
> http://talebzadehmich.wordpress.com
>
>
> *Disclaimer:* Use it at your own risk. Any and all responsibility for any
> loss, damage or destruction of data or any other property which may arise
> from relying on this email's technical content is explicitly disclaimed.
> The author will in no case be liable for any monetary damages arising from
> such loss, damage or destruction.
>
>
>
> On 1 August 2016 at 21:23, Maximiliano Patricio Méndez <
> mmendez@despegar.com> wrote:
>
>> Hi, MIch, thanks for replying.
>>
>> I'm deploying from the same instance from where I showed the logs and
>> commands using --deploy-mode cluster.
>>
>> The SparkSubmit process only appears while the bin/spark-submit binary is
>> active.
>> When the application starts and the driver takes control, the SparkSubmit
>> process dies.
>>
>> 2016-08-01 16:07 GMT-03:00 Mich Talebzadeh <mich.talebzadeh@gmail.com>:
>>
>>> OK I can see the Worker (19286 Worker and the executor(6548
>>> CoarseGrainedExecutorBackend) running on it
>>>
>>> Where is spark-submit? Did you submit your job from another node or used
>>> another method to run it?
>>>
>>> HTH
>>>
>>> Dr Mich Talebzadeh
>>>
>>>
>>>
>>> LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
>>> <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
>>>
>>>
>>>
>>> http://talebzadehmich.wordpress.com
>>>
>>>
>>> *Disclaimer:* Use it at your own risk. Any and all responsibility for
>>> any loss, damage or destruction of data or any other property which may
>>> arise from relying on this email's technical content is explicitly
>>> disclaimed. The author will in no case be liable for any monetary damages
>>> arising from such loss, damage or destruction.
>>>
>>>
>>>
>>> On 1 August 2016 at 19:08, Maximiliano Patricio Méndez <
>>> mmendez@despegar.com> wrote:
>>>
>>>> I just recently tried again, the port 4040 is not used. And even if it
>>>> were, I think the log would reflect that trying to use the following port
>>>> (4041) as you mentioned.
>>>>
>>>> This is what the driver log says:
>>>>
>>>> 16/08/01 13:55:56 INFO Utils: Successfully started service 'SparkUI' on port
4040.
>>>> 16/08/01 13:55:56 INFO SparkUI: Started SparkUI at http://hostname:4040
>>>>
>>>>
>>>> If I go to {hostname}:
>>>> ~$ jps
>>>> 6548 CoarseGrainedExecutorBackend
>>>> 19286 Worker
>>>> 6843 Jps
>>>> 19182 Master
>>>>
>>>> ~$ netstat -nltp
>>>> Active Internet connections (only servers)
>>>> Proto Recv-Q Send-Q Local Address           Foreign Address
>>>> State       PID/Program name
>>>> tcp6       0      0 192.168.22.245:43037    :::*
>>>>  LISTEN      6548/java
>>>> tcp6       0      0 192.168.22.245:56929    :::*
>>>>  LISTEN      19286/java
>>>> tcp6       0      0 192.168.22.245:7077     :::*
>>>>  LISTEN      19182/java
>>>> tcp6       0      0 :::33296                :::*
>>>>  LISTEN      6548/java
>>>> tcp6       0      0 :::8080                 :::*
>>>>  LISTEN      19182/java
>>>> tcp6       0      0 :::8081                 :::*
>>>>  LISTEN      19286/java
>>>> tcp6       0      0 192.168.22.245:6066     :::*
>>>>  LISTEN      19182/java
>>>>
>>>> ~$ netstat -nltap | grep 4040
>>>>
>>>> I'm really lost here and don't know much about spark yet, but shouldn't
>>>> there be a DriverWrapper process which holds the bind on port 4040?
>>>>
>>>>
>>>> 2016-08-01 13:49 GMT-03:00 Mich Talebzadeh <mich.talebzadeh@gmail.com>:
>>>>
>>>>> Can you check if port 4040 is actually used? If it used the next
>>>>> available one would 4041. For example below Zeppelin uses it
>>>>>
>>>>>
>>>>> *netstat -plten|grep 4040*tcp        0      0
>>>>> :::4040                     :::*                        LISTEN
>>>>> 1005       73372882   *10699*/java
>>>>> *ps aux|grep 10699*
>>>>> hduser   10699  0.1  3.8 3172308 952932 pts/3  SNl  Jul30   5:57
>>>>> /usr/java/latest/bin/java -cp /data6/hduser/zeppelin-0.6.0/ ...
>>>>>
>>>>> HTH
>>>>>
>>>>> Dr Mich Talebzadeh
>>>>>
>>>>>
>>>>>
>>>>> LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
>>>>> <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
>>>>>
>>>>>
>>>>>
>>>>> http://talebzadehmich.wordpress.com
>>>>>
>>>>>
>>>>> *Disclaimer:* Use it at your own risk. Any and all responsibility for
>>>>> any loss, damage or destruction of data or any other property which may
>>>>> arise from relying on this email's technical content is explicitly
>>>>> disclaimed. The author will in no case be liable for any monetary damages
>>>>> arising from such loss, damage or destruction.
>>>>>
>>>>>
>>>>>
>>>>> On 1 August 2016 at 17:44, Maximiliano Patricio Méndez <
>>>>> mmendez@despegar.com> wrote:
>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> Thanks for the answers.
>>>>>>
>>>>>> @Jacek: To verify if the ui is up, I enter to all the worker nodes
of
>>>>>> my cluster and run netstat -nltp | grep 4040 with no result. The
log of the
>>>>>> driver tells me in which server and on which port should the spark
ui be
>>>>>> up, but it isn't.
>>>>>>
>>>>>>
>>>>>> @Mich: I've tried to specify spark.ui.port=nnn but I only manage
to
>>>>>> change the log, reporting that the driver should be in another port.
>>>>>>
>>>>>> The ui has no problem to start in that port (4040) when I run my
>>>>>> application in client mode.
>>>>>>
>>>>>> Could there be a network issue making the ui to fail silently? I've
>>>>>> read some of the code regarding those parts of the driver log, but
couldn't
>>>>>> find anything weird.
>>>>>>
>>>>>> 2016-07-29 19:45 GMT-03:00 Mich Talebzadeh <mich.talebzadeh@gmail.com
>>>>>> >:
>>>>>>
>>>>>>> why chance it. Best to explicitly specify in spark-submit (or
>>>>>>> whatever) which port to listen to
>>>>>>>
>>>>>>>  --conf "spark.ui.port=nnn"
>>>>>>>
>>>>>>> and see if it works
>>>>>>>
>>>>>>> HTH
>>>>>>>
>>>>>>>
>>>>>>> Dr Mich Talebzadeh
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
>>>>>>> <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> http://talebzadehmich.wordpress.com
>>>>>>>
>>>>>>>
>>>>>>> *Disclaimer:* Use it at your own risk. Any and all responsibility
>>>>>>> for any loss, damage or destruction of data or any other property
which may
>>>>>>> arise from relying on this email's technical content is explicitly
>>>>>>> disclaimed. The author will in no case be liable for any monetary
damages
>>>>>>> arising from such loss, damage or destruction.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On 29 July 2016 at 23:37, Jacek Laskowski <jacek@japila.pl>
wrote:
>>>>>>>
>>>>>>>> Hi,
>>>>>>>>
>>>>>>>> I'm curious about "For some reason, sometimes the SparkUI
does not
>>>>>>>> appear to be bound on port 4040 (or any other) but the application
>>>>>>>> runs perfectly and finishes giving the expected answer."
How do you
>>>>>>>> check that web UI listens to the port 4040?
>>>>>>>>
>>>>>>>> Pozdrawiam,
>>>>>>>> Jacek Laskowski
>>>>>>>> ----
>>>>>>>> https://medium.com/@jaceklaskowski/
>>>>>>>> Mastering Apache Spark http://bit.ly/mastering-apache-spark
>>>>>>>> Follow me at https://twitter.com/jaceklaskowski
>>>>>>>>
>>>>>>>>
>>>>>>>> On Thu, Jul 28, 2016 at 11:37 PM, Maximiliano Patricio Méndez
>>>>>>>> <mmendez@despegar.com> wrote:
>>>>>>>> > Hi,
>>>>>>>> >
>>>>>>>> > I'm having some trouble trying to submit an application
to my
>>>>>>>> spark cluster.
>>>>>>>> > For some reason, sometimes the SparkUI does not appear
to be
>>>>>>>> bound on port
>>>>>>>> > 4040 (or any other) but the application runs perfectly
and
>>>>>>>> finishes giving
>>>>>>>> > the expected answer.
>>>>>>>> >
>>>>>>>> > And don't know why, but if I restart all the workers
at once
>>>>>>>> sometimes it
>>>>>>>> > begins to work and sometimes it doesn't.
>>>>>>>> >
>>>>>>>> > In the driver logs, when it fails to start the SparkUI
I see some
>>>>>>>> these
>>>>>>>> > lines:
>>>>>>>> > 16/07/28 16:13:37 INFO Utils: Successfully started service
>>>>>>>> 'SparkUI' on port
>>>>>>>> > 4040.
>>>>>>>> > 16/07/28 16:13:37 INFO SparkUI: Started SparkUI at
>>>>>>>> http://hostname-00:4040
>>>>>>>> >
>>>>>>>> > but nothing running in those ports.
>>>>>>>> >
>>>>>>>> > I'm attaching the full driver log in which I've activated
jetty
>>>>>>>> logs on
>>>>>>>> > DEBUG but couldn't find anything.
>>>>>>>> >
>>>>>>>> > The only properties that I'm not leaving at default
at the
>>>>>>>> configuration is
>>>>>>>> > the SPARK_PUBLIC_DNS=$(hostname), SPARK_WORKER_CORES
and
>>>>>>>> SPARK_WORKER_MEMORY
>>>>>>>> >
>>>>>>>> > Have anyone faced something similar?
>>>>>>>> >
>>>>>>>> > Thanks
>>>>>>>> >
>>>>>>>> >
>>>>>>>> >
>>>>>>>> ---------------------------------------------------------------------
>>>>>>>> > To unsubscribe e-mail: user-unsubscribe@spark.apache.org
>>>>>>>>
>>>>>>>>
>>>>>>>> ---------------------------------------------------------------------
>>>>>>>> To unsubscribe e-mail: user-unsubscribe@spark.apache.org
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>

Mime
View raw message