spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From DB Tsai <dbt...@dbtsai.com>
Subject Re: Disabling SparkContext WebUI on port 4040, accessing information programatically?
Date Tue, 08 Jul 2014 17:04:08 GMT
We're doing similar thing to lunch spark job in tomcat, and I opened a
JIRA for this. There are couple technical discussions there.

https://issues.apache.org/jira/browse/SPARK-2100

In this end, we realized that spark uses jetty not only for Spark
WebUI, but also for distributing the jars and tasks, so it really hard
to remove the web dependency in Spark.

In the end, we lunch our spark job in yarn-cluster mode, and in the
runtime, the only dependency in our web application is spark-yarn
which doesn't contain any spark web stuff.

PS, upgrading the spark jetty 8.x to 9.x in spark may not be
straightforward by just changing the version in spark build script.
Jetty 9.x required Java 7 since the servlet api (servlet 3.1) requires
Java 7.

Sincerely,

DB Tsai
-------------------------------------------------------
My Blog: https://www.dbtsai.com
LinkedIn: https://www.linkedin.com/in/dbtsai


On Tue, Jul 8, 2014 at 8:43 AM, Koert Kuipers <koert@tresata.com> wrote:
> do you control your cluster and spark deployment? if so, you can try to
> rebuild with jetty 9.x
>
>
> On Tue, Jul 8, 2014 at 9:39 AM, Martin Gammelsæter
> <martingammelsaeter@gmail.com> wrote:
>>
>> Digging a bit more I see that there is yet another jetty instance that
>> is causing the problem, namely the BroadcastManager has one. I guess
>> this one isn't very wise to disable... It might very well be that the
>> WebUI is a problem as well, but I guess the code doesn't get far
>> enough. Any ideas on how to solve this? Spark seems to use jetty
>> 8.1.14, while dropwizard uses jetty 9.0.7, so that might be the source
>> of the problem. Any ideas?
>>
>> On Tue, Jul 8, 2014 at 2:58 PM, Martin Gammelsæter
>> <martingammelsaeter@gmail.com> wrote:
>> > Hi!
>> >
>> > I am building a web frontend for a Spark app, allowing users to input
>> > sql/hql and get results back. When starting a SparkContext from within
>> > my server code (using jetty/dropwizard) I get the error
>> >
>> > java.lang.NoSuchMethodError:
>> > org.eclipse.jetty.server.AbstractConnector: method <init>()V not found
>> >
>> > when Spark tries to fire up its own jetty server. This does not happen
>> > when running the same code without my web server. This is probably
>> > fixable somehow(?) but I'd like to disable the webUI as I don't need
>> > it, and ideally I would like to access that information
>> > programatically instead, allowing me to embed it in my own web
>> > application.
>> >
>> > Is this possible?
>> >
>> > --
>> > Best regards,
>> > Martin Gammelsæter
>>
>>
>>
>> --
>> Mvh.
>> Martin Gammelsæter
>> 92209139
>
>

Mime
View raw message