spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Cédric Hernalsteens (JIRA) <>
Subject [jira] [Commented] (SPARK-15487) Spark Master UI to reverse proxy Application and Workers UI
Date Sun, 23 Oct 2016 21:55:58 GMT


Cédric Hernalsteens commented on SPARK-15487:

While it works great to proxy to the slave nodes (running in docker containers with ports
not exposed), there seem to be some inconsistencies in the way the URI are generated.

I specified

- SPARK_MASTER_OPTS=-Dspark.ui.reverseProxyUrl=http://dev-machine-1/spark/ -Dspark.ui.reverseProxy=true
- SPARK_WORKER_OPTS=-Dspark.ui.reverseProxyUrl=http://dev-machine-1/spark/ -Dspark.ui.reverseProxy=true

Hoping to be able to reverse proxy the whole thing with nginx:

    location /spark/ {
      proxy_pass http://spark-master:8080/;
      proxy_set_header X-Forwarded-Host $host;
      proxy_set_header X-Forwarded-Server $host;
      proxy_set_header X-Real-IP $remote_addr;
      proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;

However in the Master webUI, the URI for the slave nodes are http://dev-machine-1/proxy/...
and not http://dev-machine-1/spark/proxy/...

Then I decided to use a nginx directive  " location /proxy/ " which is a bit ugly but then
on the Worker webUI, the URI "back to master" is http://dev-machine-1/spark , apparently taking
spark.ui.reverseProxyUrl into account.

Would someone be able to point to what I'm doing wrong, or confirm this is an issue?

> Spark Master UI to reverse proxy Application and Workers UI
> -----------------------------------------------------------
>                 Key: SPARK-15487
>                 URL:
>             Project: Spark
>          Issue Type: Improvement
>          Components: Web UI
>    Affects Versions: 1.6.0, 1.6.1
>            Reporter: Gurvinder
>            Assignee: Gurvinder
>            Priority: Minor
>             Fix For: 2.1.0
> Currently when running in Standalone mode, Spark UI's link to workers and application
drivers are pointing to internal/protected network endpoints. So to access workers/application
UI user's machine has to connect to VPN or need to have access to internal network directly.
> Therefore the proposal is to make Spark master UI reverse proxy this information back
to the user. So only Spark master UI needs to be opened up to internet. 
> The minimal changes can be done by adding another route e.g.<endpoint>/
so when request goes to target, ProxyServlet kicks in and takes the <endpoint> and forwards
the request to it and send response back to user.
> More information about discussions for this features can be found on this mailing list

This message was sent by Atlassian JIRA

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message