spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jey Kottalam <>
Subject Re: accessing spark ui over an ssh tunnel
Date Sat, 26 Oct 2013 17:49:35 GMT
Minor nitpick: a proxy created using the "-D" flag should be
configured as a SOCKS proxy, not an HTTP proxy.

On Fri, Oct 25, 2013 at 9:42 PM, Matei Zaharia <> wrote:
> Hey Stephen,
> SSH actually supports creating a HTTP proxy through the -D flag. Take a look at the -D
option on our spark-ec2 script for example, which just exposes the -D option of ssh. With
this feature you can do stuff like ssh -D 8088 <host> and then configure localhost:8088
as a proxy in your web browser, and browse any page accessible from the machine you've SSHed
> Matei
> On Oct 25, 2013, at 7:40 PM, Stephen Haberman <> wrote:
>> Hey,
>> The new Spark UI looks awesome. Unfortunately, it's hard for me to use
>> in a real browser (on my desktop, vs. lynx on the cluster), since our
>> (EMR) clusters are basically only accessible via SSH.
>> I can port forward the master:8080 web UI port locally, and it looks
>> great, but I can't follow any links through to the slaves. The links
>> technically use the public AWS host names, but we don't have those
>> ports opened in the security group.
>> Two things come to mind:
>> 1) Maybe the entire UI could be rendered on the master? Either based on
>> the master's data or making actor/JSON/whatever calls to the slaves.
>> 2) Somehow proxy links through the master to each slave. Like
>> master:8080/slave/<id>/<some-path> ends up proxied to the
>> slave:8081/<some-path>, and the master just sends the HTML as-is back
>> to the client.
>> I'm not very familiar with the web UI, so I'm not sure which would be
>> easier. I also don't mean to bikeshed on the really great UI.
>> But, at least with our current setup, this would demonstrably increase
>> the usability for me.
>> Is this something that you guys have considered? Do you have any
>> thoughts on it? WIP? Open to contributions?
>> Thanks!
>> - Stephen

View raw message