spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From John Omernik <j...@omernik.com>
Subject Re: Spark 1.3.1 On Mesos Issues.
Date Thu, 04 Jun 2015 13:55:21 GMT
So a few updates.  When I run local as stated before, it works fine. When I
run in Yarn (via Apache Myriad on Mesos) it also runs fine. The only issue
is specifically with Mesos. I wonder if there is some sort of class path
goodness I need to fix or something along that lines.  Any tips would be
appreciated.

Thanks!

John

On Mon, Jun 1, 2015 at 6:14 PM, Dean Wampler <deanwampler@gmail.com> wrote:

> It would be nice to see the code for MapR FS Java API, but my google foo
> failed me (assuming it's open source)...
>
> So, shooting in the dark ;) there are a few things I would check, if you
> haven't already:
>
> 1. Could there be 1.2 versions of some Spark jars that get picked up at
> run time (but apparently not in local mode) on one or more nodes? (Side
> question: Does your node experiment fail on all nodes?) Put another way,
> are the classpaths good for all JVM tasks?
> 2. Can you use just MapR and Spark 1.3.1 successfully, bypassing Mesos?
>
> Incidentally, how are you combining Mesos and MapR? Are you running Spark
> in Mesos, but accessing data in MapR-FS?
>
> Perhaps the MapR "shim" library doesn't support Spark 1.3.1.
>
> HTH,
>
> dean
>
> Dean Wampler, Ph.D.
> Author: Programming Scala, 2nd Edition
> <http://shop.oreilly.com/product/0636920033073.do> (O'Reilly)
> Typesafe <http://typesafe.com>
> @deanwampler <http://twitter.com/deanwampler>
> http://polyglotprogramming.com
>
> On Mon, Jun 1, 2015 at 2:49 PM, John Omernik <john@omernik.com> wrote:
>
>> All -
>>
>> I am facing and odd issue and I am not really sure where to go for
>> support at this point.  I am running MapR which complicates things as it
>> relates to Mesos, however this HAS worked in the past with no issues so I
>> am stumped here.
>>
>> So for starters, here is what I am trying to run. This is a simple show
>> tables using the Hive Context:
>>
>> from pyspark import SparkContext, SparkConf
>> from pyspark.sql import SQLContext, Row, HiveContext
>> sparkhc = HiveContext(sc)
>> test = sparkhc.sql("show tables")
>> for r in test.collect():
>>   print r
>>
>> When I run it on 1.3.1 using ./bin/pyspark --master local  This works
>> with no issues.
>>
>> When I run it using Mesos with all the settings configured (as they had
>> worked in the past) I get lost tasks and when I zoom in them, the error
>> that is being reported is below.  Basically it's a NullPointerException on
>> the com.mapr.fs.ShimLoader.  What's weird to me is is I took each instance
>> and compared both together, the class path, everything is exactly the same.
>> Yet running in local mode works, and running in mesos fails.  Also of note,
>> when the task is scheduled to run on the same node as when I run locally,
>> that fails too! (Baffling).
>>
>> Ok, for comparison, how I configured Mesos was to download the mapr4
>> package from spark.apache.org.  Using the exact same configuration file
>> (except for changing the executor tgz from 1.2.0 to 1.3.1) from the 1.2.0.
>> When I run this example with the mapr4 for 1.2.0 there is no issue in
>> Mesos, everything runs as intended. Using the same package for 1.3.1 then
>> it fails.
>>
>> (Also of note, 1.2.1 gives a 404 error, 1.2.2 fails, and 1.3.0 fails as
>> well).
>>
>> So basically When I used 1.2.0 and followed a set of steps, it worked on
>> Mesos and 1.3.1 fails.  Since this is a "current" version of Spark, MapR is
>> supports 1.2.1 only.  (Still working on that).
>>
>> I guess I am at a loss right now on why this would be happening, any
>> pointers on where I could look or what I could tweak would be greatly
>> appreciated. Additionally, if there is something I could specifically draw
>> to the attention of MapR on this problem please let me know, I am perplexed
>> on the change from 1.2.0 to 1.3.1.
>>
>> Thank you,
>>
>> John
>>
>>
>>
>>
>> Full Error on 1.3.1 on Mesos:
>> 15/05/19 09:31:26 INFO MemoryStore: MemoryStore started with capacity
>> 1060.3 MB java.lang.NullPointerException at
>> com.mapr.fs.ShimLoader.getRootClassLoader(ShimLoader.java:96) at
>> com.mapr.fs.ShimLoader.injectNativeLoader(ShimLoader.java:232) at
>> com.mapr.fs.ShimLoader.load(ShimLoader.java:194) at
>> org.apache.hadoop.conf.CoreDefaultProperties.(CoreDefaultProperties.java:60)
>> at java.lang.Class.forName0(Native Method) at
>> java.lang.Class.forName(Class.java:274) at
>> org.apache.hadoop.conf.Configuration.getClassByNameOrNull(Configuration.java:1847)
>> at
>> org.apache.hadoop.conf.Configuration.getProperties(Configuration.java:2062)
>> at
>> org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:2272)
>> at
>> org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:2224)
>> at org.apache.hadoop.conf.Configuration.getProps(Configuration.java:2141)
>> at org.apache.hadoop.conf.Configuration.set(Configuration.java:992) at
>> org.apache.hadoop.conf.Configuration.set(Configuration.java:966) at
>> org.apache.spark.deploy.SparkHadoopUtil.newConfiguration(SparkHadoopUtil.scala:98)
>> at org.apache.spark.deploy.SparkHadoopUtil.(SparkHadoopUtil.scala:43) at
>> org.apache.spark.deploy.SparkHadoopUtil$.(SparkHadoopUtil.scala:220) at
>> org.apache.spark.deploy.SparkHadoopUtil$.(SparkHadoopUtil.scala) at
>> org.apache.spark.util.Utils$.getSparkOrYarnConfig(Utils.scala:1959) at
>> org.apache.spark.storage.BlockManager.(BlockManager.scala:104) at
>> org.apache.spark.storage.BlockManager.(BlockManager.scala:179) at
>> org.apache.spark.SparkEnv$.create(SparkEnv.scala:310) at
>> org.apache.spark.SparkEnv$.createExecutorEnv(SparkEnv.scala:186) at
>> org.apache.spark.executor.MesosExecutorBackend.registered(MesosExecutorBackend.scala:70)
>> java.lang.RuntimeException: Failure loading MapRClient. at
>> com.mapr.fs.ShimLoader.injectNativeLoader(ShimLoader.java:283) at
>> com.mapr.fs.ShimLoader.load(ShimLoader.java:194) at
>> org.apache.hadoop.conf.CoreDefaultProperties.(CoreDefaultProperties.java:60)
>> at java.lang.Class.forName0(Native Method) at
>> java.lang.Class.forName(Class.java:274) at
>> org.apache.hadoop.conf.Configuration.getClassByNameOrNull(Configuration.java:1847)
>> at
>> org.apache.hadoop.conf.Configuration.getProperties(Configuration.java:2062)
>> at
>> org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:2272)
>> at
>> org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:2224)
>> at org.apache.hadoop.conf.Configuration.getProps(Configuration.java:2141)
>> at org.apache.hadoop.conf.Configuration.set(Configuration.java:992) at
>> org.apache.hadoop.conf.Configuration.set(Configuration.java:966) at
>> org.apache.spark.deploy.SparkHadoopUtil.newConfiguration(SparkHadoopUtil.scala:98)
>> at org.apache.spark.deploy.SparkHadoopUtil.(SparkHadoopUtil.scala:43) at
>> org.apache.spark.deploy.SparkHadoopUtil$.(SparkHadoopUtil.scala:220) at
>> org.apache.spark.deploy.SparkHadoopUtil$.(SparkHadoopUtil.scala) at
>> org.apache.spark.util.Utils$.getSparkOrYarnConfig(Utils.scala:1959) at
>> org.apache.spark.storage.BlockManager.(BlockManager.scala:104) at
>> org.apache.spark.storage.BlockManager.(BlockManager.scala:179) at
>> org.apache.spark.SparkEnv$.create(SparkEnv.scala:310) at
>> org.apache.spark.SparkEnv$.createExecutorEnv(SparkEnv.scala:186) at
>> org.apache.spark.executor.MesosExecutorBackend.registered(MesosExecutorBackend.scala:70)
>>
>>
>>
>

Mime
View raw message