spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Martin Gammelsæter <martingammelsae...@gmail.com>
Subject Re: Disabling SparkContext WebUI on port 4040, accessing information programatically?
Date Wed, 09 Jul 2014 08:24:42 GMT
Thanks for your input, Koert and DB. Rebuilding with 9.x didn't seem
to work. For now we've downgraded dropwizard to 0.6.2 which uses a
compatible version of jetty. Not optimal, but it works for now.

On Tue, Jul 8, 2014 at 7:04 PM, DB Tsai <dbtsai@dbtsai.com> wrote:
> We're doing similar thing to lunch spark job in tomcat, and I opened a
> JIRA for this. There are couple technical discussions there.
>
> https://issues.apache.org/jira/browse/SPARK-2100
>
> In this end, we realized that spark uses jetty not only for Spark
> WebUI, but also for distributing the jars and tasks, so it really hard
> to remove the web dependency in Spark.
>
> In the end, we lunch our spark job in yarn-cluster mode, and in the
> runtime, the only dependency in our web application is spark-yarn
> which doesn't contain any spark web stuff.
>
> PS, upgrading the spark jetty 8.x to 9.x in spark may not be
> straightforward by just changing the version in spark build script.
> Jetty 9.x required Java 7 since the servlet api (servlet 3.1) requires
> Java 7.
>
> Sincerely,
>
> DB Tsai
> -------------------------------------------------------
> My Blog: https://www.dbtsai.com
> LinkedIn: https://www.linkedin.com/in/dbtsai
>
>
> On Tue, Jul 8, 2014 at 8:43 AM, Koert Kuipers <koert@tresata.com> wrote:
>> do you control your cluster and spark deployment? if so, you can try to
>> rebuild with jetty 9.x
>>
>>
>> On Tue, Jul 8, 2014 at 9:39 AM, Martin Gammelsæter
>> <martingammelsaeter@gmail.com> wrote:
>>>
>>> Digging a bit more I see that there is yet another jetty instance that
>>> is causing the problem, namely the BroadcastManager has one. I guess
>>> this one isn't very wise to disable... It might very well be that the
>>> WebUI is a problem as well, but I guess the code doesn't get far
>>> enough. Any ideas on how to solve this? Spark seems to use jetty
>>> 8.1.14, while dropwizard uses jetty 9.0.7, so that might be the source
>>> of the problem. Any ideas?
>>>
>>> On Tue, Jul 8, 2014 at 2:58 PM, Martin Gammelsæter
>>> <martingammelsaeter@gmail.com> wrote:
>>> > Hi!
>>> >
>>> > I am building a web frontend for a Spark app, allowing users to input
>>> > sql/hql and get results back. When starting a SparkContext from within
>>> > my server code (using jetty/dropwizard) I get the error
>>> >
>>> > java.lang.NoSuchMethodError:
>>> > org.eclipse.jetty.server.AbstractConnector: method <init>()V not found
>>> >
>>> > when Spark tries to fire up its own jetty server. This does not happen
>>> > when running the same code without my web server. This is probably
>>> > fixable somehow(?) but I'd like to disable the webUI as I don't need
>>> > it, and ideally I would like to access that information
>>> > programatically instead, allowing me to embed it in my own web
>>> > application.
>>> >
>>> > Is this possible?
>>> >
>>> > --
>>> > Best regards,
>>> > Martin Gammelsæter
>>>
>>>
>>>
>>> --
>>> Mvh.
>>> Martin Gammelsæter
>>> 92209139
>>
>>



-- 
Mvh.
Martin Gammelsæter
92209139

Mime
View raw message