spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Rychnovsky, Dusan" <Dusan.Rychnov...@firma.seznam.cz>
Subject Application not showing in Spark History
Date Tue, 02 Aug 2016 08:53:58 GMT
Hi,


I am trying to launch my Spark application from within my Java application via the SparkSubmit
class, like this:


List<String> args = new ArrayList<>();

args.add("--verbose");
args.add("--deploy-mode=cluster");
args.add("--master=yarn");
...


SparkSubmit.main(args.toArray(new String[args.size()]));



This works fine, with one catch - the application does not appear in Spark History after it's
finished.


If, however, I run the application using `spark-submit.sh`, like this:



spark-submit \

  --verbose \

  --deploy-mode=cluster \

  --master=yarn \

  ...



the application appears in Spark History correctly.


What am I missing?


Also, is this a good way to launch a Spark application from within a Java application or is
there a better way?

Thanks,

Dusan


Mime
View raw message