spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Maximiliano Patricio Méndez <mmen...@despegar.com>
Subject Re: Problems initializing SparkUI
Date Mon, 01 Aug 2016 16:44:14 GMT
Hi,

Thanks for the answers.

@Jacek: To verify if the ui is up, I enter to all the worker nodes of my
cluster and run netstat -nltp | grep 4040 with no result. The log of the
driver tells me in which server and on which port should the spark ui be
up, but it isn't.


@Mich: I've tried to specify spark.ui.port=nnn but I only manage to change
the log, reporting that the driver should be in another port.

The ui has no problem to start in that port (4040) when I run my
application in client mode.

Could there be a network issue making the ui to fail silently? I've read
some of the code regarding those parts of the driver log, but couldn't find
anything weird.

2016-07-29 19:45 GMT-03:00 Mich Talebzadeh <mich.talebzadeh@gmail.com>:

> why chance it. Best to explicitly specify in spark-submit (or whatever)
> which port to listen to
>
>  --conf "spark.ui.port=nnn"
>
> and see if it works
>
> HTH
>
>
> Dr Mich Talebzadeh
>
>
>
> LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
> <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
>
>
>
> http://talebzadehmich.wordpress.com
>
>
> *Disclaimer:* Use it at your own risk. Any and all responsibility for any
> loss, damage or destruction of data or any other property which may arise
> from relying on this email's technical content is explicitly disclaimed.
> The author will in no case be liable for any monetary damages arising from
> such loss, damage or destruction.
>
>
>
> On 29 July 2016 at 23:37, Jacek Laskowski <jacek@japila.pl> wrote:
>
>> Hi,
>>
>> I'm curious about "For some reason, sometimes the SparkUI does not
>> appear to be bound on port 4040 (or any other) but the application
>> runs perfectly and finishes giving the expected answer." How do you
>> check that web UI listens to the port 4040?
>>
>> Pozdrawiam,
>> Jacek Laskowski
>> ----
>> https://medium.com/@jaceklaskowski/
>> Mastering Apache Spark http://bit.ly/mastering-apache-spark
>> Follow me at https://twitter.com/jaceklaskowski
>>
>>
>> On Thu, Jul 28, 2016 at 11:37 PM, Maximiliano Patricio Méndez
>> <mmendez@despegar.com> wrote:
>> > Hi,
>> >
>> > I'm having some trouble trying to submit an application to my spark
>> cluster.
>> > For some reason, sometimes the SparkUI does not appear to be bound on
>> port
>> > 4040 (or any other) but the application runs perfectly and finishes
>> giving
>> > the expected answer.
>> >
>> > And don't know why, but if I restart all the workers at once sometimes
>> it
>> > begins to work and sometimes it doesn't.
>> >
>> > In the driver logs, when it fails to start the SparkUI I see some these
>> > lines:
>> > 16/07/28 16:13:37 INFO Utils: Successfully started service 'SparkUI' on
>> port
>> > 4040.
>> > 16/07/28 16:13:37 INFO SparkUI: Started SparkUI at
>> http://hostname-00:4040
>> >
>> > but nothing running in those ports.
>> >
>> > I'm attaching the full driver log in which I've activated jetty logs on
>> > DEBUG but couldn't find anything.
>> >
>> > The only properties that I'm not leaving at default at the
>> configuration is
>> > the SPARK_PUBLIC_DNS=$(hostname), SPARK_WORKER_CORES and
>> SPARK_WORKER_MEMORY
>> >
>> > Have anyone faced something similar?
>> >
>> > Thanks
>> >
>> >
>> > ---------------------------------------------------------------------
>> > To unsubscribe e-mail: user-unsubscribe@spark.apache.org
>>
>> ---------------------------------------------------------------------
>> To unsubscribe e-mail: user-unsubscribe@spark.apache.org
>>
>>
>

Mime
View raw message