spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Matei Zaharia <matei.zaha...@gmail.com>
Subject Re: spark 0.8
Date Fri, 18 Oct 2013 16:34:07 GMT
Can you post the build.sbt for your program? It needs to include hadoop-client for CDH4.3,
and that should *not* be listed as provided.

Matei

On Oct 18, 2013, at 8:23 AM, Koert Kuipers <koert@tresata.com> wrote:

> ok this has nothing to do with hadoop access. even a simple program that uses sc.parallelize
blows up in this way.
> 
> so spark-shell works well on the same machine i launch this from.
> 
> if i launch a simple program without using kryo for serializer and closure serialize
i get a different error. see below.
> at this point it seems to me i have some issue with task serialization???
> 
> 
> 
> 13/10/18 11:20:37 INFO StandaloneExecutorBackend: Got assigned task 0
> 13/10/18 11:20:37 INFO StandaloneExecutorBackend: Got assigned task 1
> 13/10/18 11:20:37 INFO Executor: Running task ID 1
> 13/10/18 11:20:37 INFO Executor: Running task ID 0
> 13/10/18 11:20:37 INFO Executor: Fetching http://192.168.3.171:41629/jars/simple-project_2.9.3-1.0.jar
with timestamp 1382109635095
> 13/10/18 11:20:37 INFO Utils: Fetching http://192.168.3.171:41629/jars/simple-project_2.9.3-1.0.jar
to /tmp/fetchFileTemp378181753997570700.tmp
> 13/10/18 11:20:37 INFO Executor: Adding file:/var/lib/spark/app-20131018112035-0014/1/./simple-project_2.9.3-1.0.jar
to class loader
> 13/10/18 11:20:37 INFO Executor: caught throwable with stacktrace java.io.StreamCorruptedException:
invalid type code: 00
>     at java.io.ObjectInputStream$BlockDataInputStream.readBlockHeader(ObjectInputStream.java:2467)
>     at java.io.ObjectInputStream$BlockDataInputStream.refill(ObjectInputStream.java:2502)
>     at java.io.ObjectInputStream$BlockDataInputStream.read(ObjectInputStream.java:2661)
>     at java.io.ObjectInputStream$BlockDataInputStream.read(ObjectInputStream.java:2583)
>     at java.io.DataInputStream.readFully(DataInputStream.java:178)
>     at java.io.DataInputStream.readLong(DataInputStream.java:399)
>     at java.io.ObjectInputStream$BlockDataInputStream.readLong(ObjectInputStream.java:2803)
>     at java.io.ObjectInputStream.readLong(ObjectInputStream.java:958)
>     at org.apache.spark.rdd.ParallelCollectionPartition.readObject(ParallelCollectionRDD.scala:72)
>     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>     at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>     at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>     at java.lang.reflect.Method.invoke(Method.java:597)
>     at java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:969)
>     at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1852)
>     at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1756)
>     at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1326)
>     at java.io.ObjectInputStream.readObject(ObjectInputStream.java:348)
>     at org.apache.spark.scheduler.ResultTask.readExternal(ResultTask.scala:135)
>     at java.io.ObjectInputStream.readExternalData(ObjectInputStream.java:1795)
>     at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1754)
>     at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1326)
>     at java.io.ObjectInputStream.readObject(ObjectInputStream.java:348)
>     at org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:39)
>     at org.apache.spark.serializer.JavaSerializerInstance.deserialize(JavaSerializer.scala:61)
>     at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:153)
>     at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
>     at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
>     at java.lang.Thread.run(Thread.java:662)
> 
> 
> 
> On Fri, Oct 18, 2013 at 10:59 AM, Koert Kuipers <koert@tresata.com> wrote:
> i created a tiny sbt project as described here: apache.org/docs/latest/quick-start.html#a-standalone-app-in-scala

> 
> it has the correct dependencies: spark-core and the correct hadoop-client for my platform.
i tried both the generic spark-core dependency and spark-core dependency compiled against
my platform. it runs fine in local mode, but when i switch to the cluster i still always get
the following exceptions on tasks:
> 
> 13/10/18 10:25:53 ERROR Executor: Uncaught exception in thread Thread[pool-5-thread-1,5,main]
> 
> java.lang.NullPointerException
>     at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:187)
>     at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
>     at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
>     at java.lang.Thread.run(Thread.java:662)
> 
> after adding some additional debugging to Executor i see the cause is this:
> 13/10/18 10:54:47 INFO Executor: caught throwable with stacktrace java.lang.NullPointerException
>     at org.apache.spark.executor.Executor$TaskRunner$$anonfun$run$2.apply(Executor.scala:155)
>     at org.apache.spark.executor.Executor$TaskRunner$$anonfun$run$2.apply(Executor.scala:155)
>     at org.apache.spark.Logging$class.logInfo(Logging.scala:48)
>     at org.apache.spark.executor.Executor.logInfo(Executor.scala:36)
>     at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:155)
> 
>     at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
>     at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
>     at java.lang.Thread.run(Thread.java:662)
> 
> so it seems the offending line is:
> logInfo("Its epoch is " + task.epoch)
> 
> i am guessing accessing epoch on the task is throwing the NPE. any ideas?
> 
> 
> 
> On Thu, Oct 17, 2013 at 8:12 PM, Koert Kuipers <koert@tresata.com> wrote:
> sorry one more related question:
> i compile against a spark build for hadoop 1.0.4, but the actual installed version of
spark is build against cdh4.3.0-mr1. this also used to work, and i prefer to do this so i
compile against a generic spark build. could this be the issue?
> 
> 
> On Thu, Oct 17, 2013 at 8:06 PM, Koert Kuipers <koert@tresata.com> wrote:
> i have my spark and hadoop related dependencies as "provided" for my spark job. this
used to work with previous versions. are these now supposed to be compile/runtime/default
dependencies?
> 
> 
> On Thu, Oct 17, 2013 at 8:04 PM, Koert Kuipers <koert@tresata.com> wrote:
> yes i did that and i can see the correct jars sitting in lib_managed
> 
> 
> On Thu, Oct 17, 2013 at 7:56 PM, Matei Zaharia <matei.zaharia@gmail.com> wrote:
> Koert, did you link your Spark job to the right version of HDFS as well? In Spark 0.8,
you have to add a Maven dependency on "hadoop-client" for your version of Hadoop. See http://spark.incubator.apache.org/docs/latest/quick-start.html#a-standalone-app-in-scala
for example.
> 
> Matei
> 
> On Oct 17, 2013, at 4:38 PM, Koert Kuipers <koert@tresata.com> wrote:
> 
>> i got the job a little further along by also setting this:
>> System.setProperty("spark.closure.serializer", "org.apache.spark.serializer.KryoSerializer")
>> 
>> not sure why i need to... but anyhow, now my workers start and then they blow up
on this:
>> 
>> 13/10/17 19:22:57 ERROR Executor: Uncaught exception in thread Thread[pool-5-thread-1,5,main]
>> java.lang.NullPointerException
>>     at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:187)
>>     at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
>>     at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
>>     at java.lang.Thread.run(Thread.java:662)
>> 
>> 
>> which is:
>>  val metrics = attemptedTask.flatMap(t => t.metrics)
>> 
>> 
>> 
>> 
>> 
>> 
>> 
>> 
>> 
>> On Thu, Oct 17, 2013 at 7:30 PM, dachuan <hdc1112@gmail.com> wrote:
>> thanks, Mark.
>> 
>> 
>> On Thu, Oct 17, 2013 at 6:36 PM, Mark Hamstra <mark@clearstorydata.com> wrote:
>> SNAPSHOTs are not fixed versions, but are floating names associated with whatever
is the most recent code.  So, Spark 0.8.0 is the current released version of Spark, which
is exactly the same today as it was yesterday, and will be the same thing forever.  Spark
0.8.1-SNAPSHOT is whatever is currently in branch-0.8.  It changes every time new code is
committed to that branch (which should be just bug fixes and the few additional features that
we wanted to get into 0.8.0, but that didn't quite make it.)  Not too long from now there
will be a release of Spark 0.8.1, at which time the SNAPSHOT will got to 0.8.2 and 0.8.1 will
be forever frozen.  Meanwhile, the wild new development is taking place on the master branch,
and whatever is currently in that branch becomes 0.9.0-SNAPSHOT.  This could be quite different
from day to day, and there are no guarantees that things won't be broken in 0.9.0-SNAPSHOT.
 Several months from now there will be a release of Spark 0.9.0 (unless the decision is made
to bump the version to 1.0.0), at which point the SNAPSHOT goes to 0.9.1 and the whole process
advances to the next phase of development.
>> 
>> The short answer is that releases are stable, SNAPSHOTs are not, and SNAPSHOTs that
aren't on maintenance branches can break things.  You make your choice of which to use and
pay the consequences. 
>> 
>> 
>> On Thu, Oct 17, 2013 at 3:18 PM, dachuan <hdc1112@gmail.com> wrote:
>> yeah, I mean 0.9.0-SNAPSHOT. I use git clone and that's what I got.. what's the difference?
I mean SNAPSHOT and non-SNAPSHOT.
>> 
>> 
>> On Thu, Oct 17, 2013 at 6:15 PM, Mark Hamstra <mark@clearstorydata.com> wrote:
>> Of course, you mean 0.9.0-SNAPSHOT.  There is no Spark 0.9.0, and won't be for several
months.
>> 
>> 
>> 
>> On Thu, Oct 17, 2013 at 3:11 PM, dachuan <hdc1112@gmail.com> wrote:
>> I'm sorry if this doesn't answer your question directly, but I have tried spark 0.9.0
and hdfs 1.0.4 just now, it works..
>> 
>> 
>> On Thu, Oct 17, 2013 at 6:05 PM, Koert Kuipers <koert@tresata.com> wrote:
>> after upgrading from spark 0.7 to spark 0.8 i can no longer access any files on HDFS.
>> i see the error below. any ideas?
>> 
>> i am running spark standalone on a cluster that also has CDH4.3.0 and rebuild spark
accordingly. the jars in lib_managed look good to me.
>> 
>> i noticed similar errors in the mailing list but found no suggested solutions. 
>> 
>> thanks! koert
>> 
>> 
>> 13/10/17 17:43:23 ERROR Executor: Exception in task ID 0
>> java.io.EOFException
>> 	at java.io.ObjectInputStream$BlockDataInputStream.readFully(ObjectInputStream.java:2703)
>> 	at java.io.ObjectInputStream.readFully(ObjectInputStream.java:1008)
>> 	at org.apache.hadoop.io.DataOutputBuffer$Buffer.write(DataOutputBuffer.java:68)
>> 	at org.apache.hadoop.io.DataOutputBuffer.write(DataOutputBuffer.java:106)
>> 	at org.apache.hadoop.io.UTF8.readChars(UTF8.java:258)
>> 	at org.apache.hadoop.io.UTF8.readString(UTF8.java:250)
>> 	at org.apache.hadoop.mapred.FileSplit.readFields(FileSplit.java:87)
>> 	at org.apache.hadoop.io.ObjectWritable.readObject(ObjectWritable.java:280)
>> 	at org.apache.hadoop.io.ObjectWritable.readFields(ObjectWritable.java:75)
>> 	at org.apache.spark.SerializableWritable.readObject(SerializableWritable.scala:39)
>> 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>> 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> 	at java.lang.reflect.Method.invoke(Method.java:597)
>> 	at java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:969)
>> 	at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1852)
>> 	at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1756)
>> 	at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1326)
>> 	at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1950)
>> 	at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1874)
>> 	at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1756)
>> 	at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1326)
>> 	at java.io.ObjectInputStream.readObject(ObjectInputStream.java:348)
>> 	at org.apache.spark.scheduler.ResultTask.readExternal(ResultTask.scala:135)
>> 	at java.io.ObjectInputStream.readExternalData(ObjectInputStream.java:1795)
>> 	at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1754)
>> 	at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1326)
>> 	at java.io.ObjectInputStream.readObject(ObjectInputStream.java:348)
>> 	at org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:39)
>> 	at org.apache.spark.serializer.JavaSerializerInstance.deserialize(JavaSerializer.scala:61)
>> 	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:153)
>> 	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
>> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
>> 	at java.lang.Thread.run(Thread.java:662)
>> 
>> 
>> 
>> -- 
>> Dachuan Huang
>> Cellphone: 614-390-7234
>> 2015 Neil Avenue
>> Ohio State University
>> Columbus, Ohio
>> U.S.A.
>> 43210
>> 
>> 
>> 
>> 
>> -- 
>> Dachuan Huang
>> Cellphone: 614-390-7234
>> 2015 Neil Avenue
>> Ohio State University
>> Columbus, Ohio
>> U.S.A.
>> 43210
>> 
>> 
>> 
>> 
>> -- 
>> Dachuan Huang
>> Cellphone: 614-390-7234
>> 2015 Neil Avenue
>> Ohio State University
>> Columbus, Ohio
>> U.S.A.
>> 43210
>> 
> 
> 
> 
> 
> 
> 


Mime
View raw message