spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Carl Ballantyne <carl.ballant...@guvera.com>
Subject Re: Spark Standalone Cluster - Running applications in JSON format
Date Thu, 01 Dec 2016 05:59:19 GMT
Hmmm getting closer I think.

I thought this was only for Mesos and Yarn clusters (from reading the 
documentation). I tried anyway and initially received Connection 
Refused. So I ran ./start-history-server.sh. This was on the Spark 
Master instance.

I now get 404 not found.

Nothing in the log file for the history server indicates there was a 
problem.

I will keep digging around. Thanks for your help so far Miguel.


On 1/12/2016 3:33 PM, Miguel Morales wrote:
> Try hitting:  http://<server-url>:18080/api/v1
>
> Then hit /applications.
>
> That should give you a list of running spark jobs on a given server.
>
> On Wed, Nov 30, 2016 at 9:30 PM, Carl Ballantyne
> <carl.ballantyne@guvera.com> wrote:
>> Yes I was looking at this. But it says I need to access the driver - http://<driver-node>:4040.
>>
>> I don't have a running driver Spark instance since I am submitting jobs to Spark
using the SparkLauncher class. Or maybe I am missing something obvious. Apologies if so.
>>
>>
>>
>>
>> On 1/12/2016 3:21 PM, Miguel Morales wrote:
>>
>> Check the Monitoring and Instrumentation API: http://spark.apache.org/docs/latest/monitoring.html
>>
>> On Wed, Nov 30, 2016 at 9:20 PM, Carl Ballantyne <carl.ballantyne@guvera.com>
wrote:
>>> Hi All,
>>>
>>> I want to get the running applications for my Spark Standalone cluster in JSON
format. The same information displayed on the web UI on port 8080 ... but in JSON.
>>>
>>> Is there an easy way to do this? It seems I need to scrap the HTML page in order
to get this information.
>>>
>>> The reason I want to know this information is so I can ensure the Spark cluster
does not get too many jobs submitted at once. A Stand alone cluster processes jobs FIFO. I
would prefer to just send back a message to the user telling them to try later then submit
a job which has to wait for other jobs to finish before starting.
>>>
>>> Any help appreciated. Thanks.
>>>
>>> Cheers,
>>> Carl
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe e-mail: user-unsubscribe@spark.apache.org
>>>
>>
>> --
>> Carl Ballantyne
>> Lead Reporting Developer
>> Guvera Operations Pty Ltd.
>> Suite 1b, 58 Kingston Drive
>> Helensvale, QLD, 4212
>> Australia
>> PO Box 3330
>> Helensvale Town Centre, QLD, 4212
>> Phone +61 (0) 7 5578 8987
>> Email carl.ballantyne@guvera.com
>> Web www.guveralimited.com
>>
>>

-- 
*Carl Ballantyne*
Lead Reporting Developer
*Guvera Operations Pty Ltd.*
Suite 1b, 58 Kingston Drive
Helensvale, QLD, 4212
Australia
PO Box 3330
Helensvale Town Centre, QLD, 4212
*Phone *+61 (0) 7 5578 8987
*Email *carl.ballantyne@guvera.com <mailto:carl.ballantyne@guvera.com>
*Web *www.guveralimited.com <http://www.guveralimited.com/>



Mime
View raw message