spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From ", Roy" <rp...@njit.edu>
Subject Re: Spark error NoClassDefFoundError: org/apache/hadoop/mapred/InputSplit
Date Mon, 23 Mar 2015 18:40:14 GMT
now job started but it stuck at some different level

15/03/23 14:09:00 INFO BlockManagerInfo: Added rdd_10_0 on disk on
ip-10-0-3-171.ec2.internal:58704 (size: 138.8 MB)

so i checked yarn logs on ip-10-0-3-171.ec2.internal but I dint see any
errors ?

anyone know whats going on here ?



*Thanks...*

*Roy*

On Mon, Mar 23, 2015 at 12:13 PM, Ted Yu <yuzhihong@gmail.com> wrote:

> InputSplit is in hadoop-mapreduce-client-core jar
>
> Please check that the jar is in your classpath.
>
> Cheers
>
> On Mon, Mar 23, 2015 at 8:10 AM, , Roy <rp346@njit.edu> wrote:
>
>> Hi,
>>
>>
>>   I am using CDH 5.3.2 packages installation through Cloudera Manager
>> 5.3.2
>>
>> I am trying to run one spark job with following command
>>
>> PYTHONPATH=~/code/utils/ spark-submit --master yarn --executor-memory 3G
>> --num-executors 30 --driver-memory 2G --executor-cores 2 --name=analytics
>> /home/abc/code/updb/spark/UPDB3analytics.py -date 2015-03-01
>>
>> but I am getting following error
>>
>> 15/03/23 11:06:49 WARN TaskSetManager: Lost task 9.0 in stage 0.0 (TID 7,
>> hdp003.dev.xyz.com): java.lang.NoClassDefFoundError:
>> org/apache/hadoop/mapred/InputSplit
>> at java.lang.Class.getDeclaredConstructors0(Native Method)
>> at java.lang.Class.privateGetDeclaredConstructors(Class.java:2532)
>> at java.lang.Class.getDeclaredConstructors(Class.java:1901)
>> at
>> java.io.ObjectStreamClass.computeDefaultSUID(ObjectStreamClass.java:1749)
>> at java.io.ObjectStreamClass.access$100(ObjectStreamClass.java:72)
>> at java.io.ObjectStreamClass$1.run(ObjectStreamClass.java:250)
>> at java.io.ObjectStreamClass$1.run(ObjectStreamClass.java:248)
>> at java.security.AccessController.doPrivileged(Native Method)
>> at
>> java.io.ObjectStreamClass.getSerialVersionUID(ObjectStreamClass.java:247)
>> at java.io.ObjectStreamClass.initNonProxy(ObjectStreamClass.java:613)
>> at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1622)
>> at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1517)
>> at
>> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1771)
>> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
>> at
>> java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1990)
>> at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1915)
>> at
>> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798)
>> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
>> at java.io.ObjectInputStream.readObject(ObjectInputStream.java:370)
>> at
>> org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:62)
>> at
>> org.apache.spark.serializer.JavaSerializerInstance.deserialize(JavaSerializer.scala:87)
>> at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:178)
>> at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>> at java.lang.Thread.run(Thread.java:745)
>> Caused by: java.lang.ClassNotFoundException:
>> org.apache.hadoop.mapred.InputSplit
>> at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
>> at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
>> at java.security.AccessController.doPrivileged(Native Method)
>> at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
>> at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
>> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
>> at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
>> ... 25 more
>>
>> here is the full trace
>>
>> https://gist.github.com/anonymous/3492f0ec63d7a23c47cf
>>
>>
>

Mime
View raw message