spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Nilkanth Patel <nilkanth.hpa...@gmail.com>
Subject Standalone Spark, How to find (driver's ) final status for an application
Date Thu, 26 Sep 2019 05:42:34 GMT
I am setting up *Spark 2.2.0 in standalone mode* (
https://spark.apache.org/docs/latest/spark-standalone.html) and submitting
spark jobs programatically using

 SparkLauncher sparkAppLauncher = new
SparkLauncher(userNameMap).setMaster(sparkMaster).setAppName(appName).....;
 SparkAppHandle sparkAppHandle = sparkAppLauncher.startApplication();

 I do have* java client program that polls Job status for the jobs
submitted programatically*, for which i am using following REST endpoint.
 curl  http://192.168.1.139:8080/json/ which provide JSON response as
following,

{
  "url" : "spark://192.168.1.139:7077",
  "workers" : [ { "id" : "x", "host" : "x", "port" : x, "webuiaddress" : "x",
                  "cores" : x,  "coresused" : x, "coresfree" : x,
"memory" : xx,
                  "memoryused" : xx,  "memoryfree" : xx,  "state" :
"x", "lastheartbeat" : x
                }, { ...},  ],
  "cores" : x,
  "coresused" : x,
  "memory" : x,
  "memoryused" : x,
  "activeapps" : [ ],
  "completedapps" : [ { "starttime" : x, "id" : "app-xx-xxxx", "name"
: "abc", "user" : "xx",
                         "memoryperslave" : x, "submitdate" :
"x","state" : "FINISHED OR RUNNING", "duration" : x
                      }, {...}],
  "activedrivers" : [ ],
  "status" : "x"}


In above response, I have observed state for *completedapps is always
FINISHED even if application fails*, while on UI (http://master:8080),
associated driver shows a failed state, like.

[image: image.png]

[image: image.png]

Referring to above example, Currently, My java client gets status  for
application (app-20190925115750-0003
<http://192.168.1.139:8080/app?appId=app-20190925115750-0003>) FINISHED,
even though it got failed (encountered exception) and associated driver
shows "FAILED" state.* I intend to show the final status in this case as
FAILED.*
It seems if i can co-relate, an application-id (app-20190925115750-0003
<http://192.168.1.139:8080/app?appId=app-20190925115750-0003>) to driver-id
(driver-20190925115748-0003), I can report a "FAILED" (final) status. I
could not find any co-relation between them (appID --> driver ID).

*Looking forward to your suggestions to resolving this or any possible
approaches to achieve this.* I have also come across some hidden REST APIs
like
http://xx.xx.xx.xx:6066/v1/submissions/status/driver-20190925115748-0003, which
seems have a limited info returned in response.


Thanks in advance.
Nilkanth.

Mime
View raw message