spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Nirav Patel <npa...@xactlycorp.com>
Subject apache-spark 1.3.0 and yarn integration and spring-boot as a container
Date Thu, 30 Jul 2015 17:47:36 GMT
Hi, I was running spark application as a query service (much like
spark-shell but within my servlet container provided by spring-boot) with
spark 1.0.2 and standalone mode. Now After upgrading to spark 1.3.1 and
trying to use Yarn instead of standalone cluster things going south for me.
I created uber jar with all dependencies (spark-core, spark-yarn,
spring-boot) and tried to deploy my application.

    15/07/29 11:19:26 INFO client.RMProxy: Connecting to ResourceManager at
/0.0.0.0:8032

    15/07/29 11:19:27 INFO ipc.Client: Retrying connect to server:
0.0.0.0/0.0.0.0:8032. Already tried 0 time(s); retry policy is
RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000
MILLISECONDS)

    15/07/29 11:19:28 INFO ipc.Client: Retrying connect to server:
0.0.0.0/0.0.0.0:8032. Already tried 1 time(s); retry policy is
RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000
MILLISECONDS)

    15/07/29 11:19:29 INFO ipc.Client: Retrying connect to server:
0.0.0.0/0.0.0.0:8032. Already tried 2 time(s); retry policy is
RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000
MILLISECONDS)

I also tried to exclude spark-yarn dependencies and supplied it during
runtime but same exception. We use MapR distribution and they said it's not
possible to run spark jobs on yarn without using spark-submit script. I can
try to launch my webapp using that script as my build artifact is
spring-boot jar (not war) but that just doesn't feel right. I should be
able to init service from my container not other way around.

-- 


[image: What's New with Xactly] <http://www.xactlycorp.com/email-click/>

 <https://www.nyse.com/quote/XNYS:XTLY>  [image: LinkedIn] 
<https://www.linkedin.com/company/xactly-corporation>  [image: Twitter] 
<https://twitter.com/Xactly>  [image: Facebook] 
<https://www.facebook.com/XactlyCorp>  [image: YouTube] 
<http://www.youtube.com/xactlycorporation>

Mime
View raw message