OK I can see the Worker (19286 Worker and the executor(6548 CoarseGrainedExecutorBackend) running on it

Where is spark-submit? Did you submit your job from another node or used another method to run it?

HTH

Dr Mich Talebzadeh

 

LinkedIn  https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw

 

http://talebzadehmich.wordpress.com


Disclaimer: Use it at your own risk. Any and all responsibility for any loss, damage or destruction of data or any other property which may arise from relying on this email's technical content is explicitly disclaimed. The author will in no case be liable for any monetary damages arising from such loss, damage or destruction.

 


On 1 August 2016 at 19:08, Maximiliano Patricio Méndez <mmendez@despegar.com> wrote:
I just recently tried again, the port 4040 is not used. And even if it were, I think the log would reflect that trying to use the following port (4041) as you mentioned.

This is what the driver log says:
16/08/01 13:55:56 INFO Utils: Successfully started service 'SparkUI' on port 4040.
16/08/01 13:55:56 INFO SparkUI: Started SparkUI at http://hostname:4040

If I go to {hostname}:
~$ jps
6548 CoarseGrainedExecutorBackend
19286 Worker
6843 Jps
19182 Master

~$ netstat -nltp
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp6       0      0 192.168.22.245:43037    :::*                    LISTEN      6548/java       
tcp6       0      0 192.168.22.245:56929    :::*                    LISTEN      19286/java      
tcp6       0      0 192.168.22.245:7077     :::*                    LISTEN      19182/java      
tcp6       0      0 :::33296                :::*                    LISTEN      6548/java       
tcp6       0      0 :::8080                 :::*                    LISTEN      19182/java      
tcp6       0      0 :::8081                 :::*                    LISTEN      19286/java      
tcp6       0      0 192.168.22.245:6066     :::*                    LISTEN      19182/java      

~$ netstat -nltap | grep 4040

I'm really lost here and don't know much about spark yet, but shouldn't there be a DriverWrapper process which holds the bind on port 4040?


2016-08-01 13:49 GMT-03:00 Mich Talebzadeh <mich.talebzadeh@gmail.com>:
Can you check if port 4040 is actually used? If it used the next available one would 4041. For example below Zeppelin uses it

 netstat -plten|grep 4040
tcp        0      0 :::4040                     :::*                        LISTEN      1005       73372882   10699/java
ps aux|grep 10699
hduser   10699  0.1  3.8 3172308 952932 pts/3  SNl  Jul30   5:57 /usr/java/latest/bin/java -cp /data6/hduser/zeppelin-0.6.0/ ...

HTH

Dr Mich Talebzadeh

 

LinkedIn  https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw

 

http://talebzadehmich.wordpress.com


Disclaimer: Use it at your own risk. Any and all responsibility for any loss, damage or destruction of data or any other property which may arise from relying on this email's technical content is explicitly disclaimed. The author will in no case be liable for any monetary damages arising from such loss, damage or destruction.

 


On 1 August 2016 at 17:44, Maximiliano Patricio Méndez <mmendez@despegar.com> wrote:
Hi,

Thanks for the answers.

@Jacek: To verify if the ui is up, I enter to all the worker nodes of my cluster and run netstat -nltp | grep 4040 with no result. The log of the driver tells me in which server and on which port should the spark ui be up, but it isn't.


@Mich: I've tried to specify spark.ui.port=nnn but I only manage to change the log, reporting that the driver should be in another port.

The ui has no problem to start in that port (4040) when I run my application in client mode.

Could there be a network issue making the ui to fail silently? I've read some of the code regarding those parts of the driver log, but couldn't find anything weird.

2016-07-29 19:45 GMT-03:00 Mich Talebzadeh <mich.talebzadeh@gmail.com>:
why chance it. Best to explicitly specify in spark-submit (or whatever) which port to listen to

 --conf "spark.ui.port=nnn"

and see if it works

HTH


Dr Mich Talebzadeh

 

LinkedIn  https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw

 

http://talebzadehmich.wordpress.com


Disclaimer: Use it at your own risk. Any and all responsibility for any loss, damage or destruction of data or any other property which may arise from relying on this email's technical content is explicitly disclaimed. The author will in no case be liable for any monetary damages arising from such loss, damage or destruction.

 


On 29 July 2016 at 23:37, Jacek Laskowski <jacek@japila.pl> wrote:
Hi,

I'm curious about "For some reason, sometimes the SparkUI does not
appear to be bound on port 4040 (or any other) but the application
runs perfectly and finishes giving the expected answer." How do you
check that web UI listens to the port 4040?

Pozdrawiam,
Jacek Laskowski
----
https://medium.com/@jaceklaskowski/
Mastering Apache Spark http://bit.ly/mastering-apache-spark
Follow me at https://twitter.com/jaceklaskowski


On Thu, Jul 28, 2016 at 11:37 PM, Maximiliano Patricio Méndez
<mmendez@despegar.com> wrote:
> Hi,
>
> I'm having some trouble trying to submit an application to my spark cluster.
> For some reason, sometimes the SparkUI does not appear to be bound on port
> 4040 (or any other) but the application runs perfectly and finishes giving
> the expected answer.
>
> And don't know why, but if I restart all the workers at once sometimes it
> begins to work and sometimes it doesn't.
>
> In the driver logs, when it fails to start the SparkUI I see some these
> lines:
> 16/07/28 16:13:37 INFO Utils: Successfully started service 'SparkUI' on port
> 4040.
> 16/07/28 16:13:37 INFO SparkUI: Started SparkUI at http://hostname-00:4040
>
> but nothing running in those ports.
>
> I'm attaching the full driver log in which I've activated jetty logs on
> DEBUG but couldn't find anything.
>
> The only properties that I'm not leaving at default at the configuration is
> the SPARK_PUBLIC_DNS=$(hostname), SPARK_WORKER_CORES and SPARK_WORKER_MEMORY
>
> Have anyone faced something similar?
>
> Thanks
>
>
> ---------------------------------------------------------------------
> To unsubscribe e-mail: user-unsubscribe@spark.apache.org

---------------------------------------------------------------------
To unsubscribe e-mail: user-unsubscribe@spark.apache.org