spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jey Kottalam <...@cs.berkeley.edu>
Subject Re: accessing spark ui over an ssh tunnel
Date Sat, 26 Oct 2013 17:49:35 GMT
Minor nitpick: a proxy created using the "-D" flag should be
configured as a SOCKS proxy, not an HTTP proxy.

On Fri, Oct 25, 2013 at 9:42 PM, Matei Zaharia <matei.zaharia@gmail.com> wrote:
> Hey Stephen,
>
> SSH actually supports creating a HTTP proxy through the -D flag. Take a look at the -D
option on our spark-ec2 script for example, which just exposes the -D option of ssh. With
this feature you can do stuff like ssh -D 8088 <host> and then configure localhost:8088
as a proxy in your web browser, and browse any page accessible from the machine you've SSHed
into.
>
> Matei
>
> On Oct 25, 2013, at 7:40 PM, Stephen Haberman <stephen.haberman@gmail.com> wrote:
>
>> Hey,
>>
>> The new Spark UI looks awesome. Unfortunately, it's hard for me to use
>> in a real browser (on my desktop, vs. lynx on the cluster), since our
>> (EMR) clusters are basically only accessible via SSH.
>>
>> I can port forward the master:8080 web UI port locally, and it looks
>> great, but I can't follow any links through to the slaves. The links
>> technically use the public AWS host names, but we don't have those
>> ports opened in the security group.
>>
>> Two things come to mind:
>>
>> 1) Maybe the entire UI could be rendered on the master? Either based on
>> the master's data or making actor/JSON/whatever calls to the slaves.
>>
>> 2) Somehow proxy links through the master to each slave. Like
>> master:8080/slave/<id>/<some-path> ends up proxied to the
>> slave:8081/<some-path>, and the master just sends the HTML as-is back
>> to the client.
>>
>> I'm not very familiar with the web UI, so I'm not sure which would be
>> easier. I also don't mean to bikeshed on the really great UI.
>>
>> But, at least with our current setup, this would demonstrably increase
>> the usability for me.
>>
>> Is this something that you guys have considered? Do you have any
>> thoughts on it? WIP? Open to contributions?
>>
>> Thanks!
>>
>> - Stephen
>

Mime
View raw message