spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From ckanth99 <>
Subject Re: Spark launcher handle and listener not giving state
Date Tue, 30 Aug 2016 02:12:54 GMT
That must be it then :(
We are using Cloudera distribution which has Spark 1.5.1.

Thanks a lot Marcelo 

---- On Mon, 29 Aug 2016 16:39:34 -0700 Marcelo Vanzin&lt;;
wrote ---- 

You haven't said which version of Spark you are using. The state API 
only works if the underlying Spark version is also 1.6 or later. 
On Mon, Aug 29, 2016 at 4:36 PM, ckanth99 &lt;; wrote: 
&gt; Hi All, 
&gt; I have a web application which will submit spark jobs on Cloudera spark 
&gt; cluster using spark launcher library. 
&gt; It is successfully submitting the spark job to cluster. However it is not 
&gt; calling back the listener class methods and also the getState() on returned 
&gt; SparkAppHandle never changes from "UNKNOWN" even after job finishes 
&gt; execution on cluster. 
&gt; I am using yarn-cluster mode. Here is my code. Is anything else needs to be 
&gt; done or is this a bug? 
&gt; SparkLauncher launcher = new SparkLauncher() 
&gt; .setSparkHome("sparkhome") 
&gt; .setMaster("yarn-cluster") .setAppResource("spark 
&gt; job jar file") .setMainClass("spark job driver 
&gt; class") .setAppName("appname") 
&gt; .addAppArgs(argsArray) .setVerbose(true) 
&gt; .addSparkArg("--verbose"); SparkAppHandle handle = 
&gt; launcher.startApplication(new LauncherListener()); int c = 0; 
&gt; while(!handle.getState().isFinal()) {"&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;
state is= 
&gt; "+handle.getState() );"&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;
state is not final yet. counter= 
&gt; "+c++ );"&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;
sleeping for a second"); try { 
&gt; Thread.sleep(1000L); } catch (InterruptedException e) { } if(c == 200) 
&gt; break; } 
&gt; Here are the things I have already tried: 
&gt; Added listener instance to SparkAppHandle once application is launched. 
&gt; Made the current class implement SparkAppHandle.Listener and passed it 
&gt; (this) in both ways (while launching, and by setting it on SparkAppHandle) 
&gt; Tried to use launcher.launch() method so that at least I can block on the 
&gt; resulting Process object by calling process.waitFor() method till spark job 
&gt; finishes running on cluster. However in this case, for long running spark 
&gt; jobs, corresponding process on this node never returns (though it works fine 
&gt; for spark jobs which are finishing in 1 or 2 min) 
&gt; Thanks, 
&gt; Reddy 

View raw message