spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sebastian Kochman <>
Subject Extending Spark REST API
Date Thu, 24 Mar 2016 20:09:34 GMT
I have a question: is there a way to extend Spark REST API with higher-level, application-specific
handlers? (exposing additional information, like app-specific metrics, but also taking some
actions within an app)
Yes, I could host my own REST endpoint within an app, but then I have to solve the problem
of advertising that endpoint to the client. The app will be deployed on YARN, so the port
cannot be fixed (since multiple AMs can run on a single machine). Spark's REST API endpoint
is already "advertised" by YARN RM to the client, so I was hoping I could just add to it.
I tried calling sparkContext.metricsSystem.getServletHandlers, but the Jetty dependencies
are shaded (I guess so the application could use a different version of Jetty), so its de-facto
signature is as follows:
    public org.spark-project.jetty.servlet.ServletContextHandler[] getServletHandlers();
(note the org.spark-project package) when I add the Jetty dependency (org.eclipse.jetty...) to my project, it compiles but
during runtime it throws:
    java.lang.NoSuchMethodError: org.apache.spark.metrics.MetricsSystem.getServletHandlers()[Lorg/eclipse/jetty/servlet/ServletContextHandler;
So it seems to me that getServletHandlers is not intended for public use - correct?
Is there any other way to extended Spark's REST API?
I would appreciate your help!
View raw message