spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Carl Ballantyne <carl.ballant...@guvera.com>
Subject Re: Spark Standalone Cluster - Running applications in JSON format
Date Thu, 01 Dec 2016 06:07:00 GMT
8080 is just the normal web UI. Which is the information I want, ie 
Running Applications, but in HTML format. I want it in JSON so I don't 
have to be scraping and parsing HTML.

 From my understanding api/v1/applications should do the trick ... 
except it doesn't.

Ah well.

On 1/12/2016 4:00 PM, Miguel Morales wrote:
> Don't have a Spark cluster up to verify this, but try port 8080.
>
> http://spark-master-ip:8080/api/v1/applications.
>
> But glad to hear you're getting somewhere, best of luck.
>
> On Wed, Nov 30, 2016 at 9:59 PM, Carl Ballantyne 
> <carl.ballantyne@guvera.com <mailto:carl.ballantyne@guvera.com>> wrote:
>
>     Hmmm getting closer I think.
>
>     I thought this was only for Mesos and Yarn clusters (from reading
>     the documentation). I tried anyway and initially received
>     Connection Refused. So I ran ./start-history-server.sh. This was
>     on the Spark Master instance.
>
>     I now get 404 not found.
>
>     Nothing in the log file for the history server indicates there was
>     a problem.
>
>     I will keep digging around. Thanks for your help so far Miguel.
>
>
>     On 1/12/2016 3:33 PM, Miguel Morales wrote:
>>     Try hitting:http://<server-url>:18080/api/v1
>>
>>     Then hit /applications.
>>
>>     That should give you a list of running spark jobs on a given server.
>>
>>     On Wed, Nov 30, 2016 at 9:30 PM, Carl Ballantyne
>>     <carl.ballantyne@guvera.com> <mailto:carl.ballantyne@guvera.com>
 wrote:
>>>     Yes I was looking at this. But it says I need to access the driver -http://<driver-node>:4040.
>>>
>>>     I don't have a running driver Spark instance since I am submitting jobs to
Spark using the SparkLauncher class. Or maybe I am missing something obvious. Apologies if
so.
>>>
>>>
>>>
>>>
>>>     On 1/12/2016 3:21 PM, Miguel Morales wrote:
>>>
>>>     Check the Monitoring and Instrumentation API:http://spark.apache.org/docs/latest/monitoring.html
>>>     <http://spark.apache.org/docs/latest/monitoring.html>
>>>
>>>     On Wed, Nov 30, 2016 at 9:20 PM, Carl Ballantyne<carl.ballantyne@guvera.com>
<mailto:carl.ballantyne@guvera.com>  wrote:
>>>>     Hi All,
>>>>
>>>>     I want to get the running applications for my Spark Standalone cluster
in JSON format. The same information displayed on the web UI on port 8080 ... but in JSON.
>>>>
>>>>     Is there an easy way to do this? It seems I need to scrap the HTML page
in order to get this information.
>>>>
>>>>     The reason I want to know this information is so I can ensure the Spark
cluster does not get too many jobs submitted at once. A Stand alone cluster processes jobs
FIFO. I would prefer to just send back a message to the user telling them to try later then
submit a job which has to wait for other jobs to finish before starting.
>>>>
>>>>     Any help appreciated. Thanks.
>>>>
>>>>     Cheers,
>>>>     Carl
>>>>
>>>>     ---------------------------------------------------------------------
>>>>     To unsubscribe e-mail:user-unsubscribe@spark.apache.org
>>>>     <mailto:user-unsubscribe@spark.apache.org>
>>>>
>>>     --
>>>     Carl Ballantyne
>>>     Lead Reporting Developer
>>>     Guvera Operations Pty Ltd.
>>>     Suite 1b, 58 Kingston Drive
>>>     Helensvale, QLD, 4212
>>>     Australia
>>>     PO Box 3330
>>>     Helensvale Town Centre, QLD, 4212
>>>     Phone+61 (0) 7 5578 8987 <tel:+61%207%205578%208987>
>>>     Emailcarl.ballantyne@guvera.com <mailto:carl.ballantyne@guvera.com>
>>>     Webwww.guveralimited.com <http://www.guveralimited.com>
>>>
>>>
>     -- 
>     *Carl Ballantyne* Lead Reporting Developer
>     *Guvera Operations Pty Ltd.* Suite 1b, 58 Kingston Drive
>     Helensvale, QLD, 4212 Australia
>     PO Box 3330 Helensvale Town Centre, QLD, 4212
>     *Phone *+61 (0) 7 5578 8987 <tel:+61%207%205578%208987> *Email
>     *carl.ballantyne@guvera.com <mailto:carl.ballantyne@guvera.com>
>     *Web *www.guveralimited.com <http://www.guveralimited.com/>
>
-- 
*Carl Ballantyne* Lead Reporting Developer
*Guvera Operations Pty Ltd.* Suite 1b, 58 Kingston Drive Helensvale, 
QLD, 4212 Australia
PO Box 3330 Helensvale Town Centre, QLD, 4212
*Phone *+61 (0) 7 5578 8987 *Email *carl.ballantyne@guvera.com 
<mailto:carl.ballantyne@guvera.com> *Web *www.guveralimited.com 
<http://www.guveralimited.com/>


Mime
View raw message