spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Evan Chan ...@ooyala.com>
Subject Re: Getting failures in FileServerSuite
Date Mon, 04 Nov 2013 00:05:14 GMT
Oops, hit enter too soon.

Mark: 1.6.0_51.   I'm hesitant to upgrade to JDK 1.7, as several folks
reported problems on OSX.   Also, I have no problems building the
assembly, and it only takes me about 2 minutes (I'm running on SSD's
though  :)

I might bite the bullet and upgrade to JDK 1.7 for other reasons though.

Patrick:  I know the branch with my config overhaul, last merged from
master Oct-10th, doesn't exhibit this problem.  (Note that I tend to
set SPARK_LOCAL_IP to "localhost", which I don't think affects this,
and it fails either with it set or not, i believe)  We could
potentially run a git bisect starting roughly 2-3 weeks ago.



On Sun, Nov 3, 2013 at 3:55 PM, Evan Chan <ev@ooyala.com> wrote:
> Mark:  I'm using JDK 1.6. --
>
> On Wed, Oct 30, 2013 at 1:44 PM, Mark Hamstra <mark@clearstorydata.com> wrote:
>> What JDK version on you using, Evan?
>>
>> I tried to reproduce your problem earlier today, but I wasn't even able to
>> get through the assembly build -- kept hanging when trying to build the
>> examples assembly.  Foregoing the assembly and running the tests would hang
>> on FileServerSuite "Dynamically adding JARS locally" -- no stack trace,
>> just hung.  And I was actually seeing a very similar stack trace to yours
>> from a test suite of our own running against 0.8.1-SNAPSHOT -- not exactly
>> the same because line numbers were different once it went into the java
>> runtime, and it eventually ended up someplace a little different.  That got
>> me curious about differences in Java versions, so I updated to the latest
>> Oracle release (1.7.0_45).  Now it cruises right through the build and test
>> of Spark master from before Matei merged your PR.  Then I logged into a
>> machine that has 1.7.0_15 (7u15-2.3.7-0ubuntu1~11.10.1, actually)
>> installed, and I'm right back to the hanging during the examples assembly
>> (but passes FileServerSuite, oddly enough.)  Upgrading the JDK didn't
>> improve the results of the ClearStory test suite I was looking at, so my
>> misery isn't over; but yours might be with a newer JDK....
>>
>>
>>
>> On Wed, Oct 30, 2013 at 12:44 PM, Evan Chan <ev@ooyala.com> wrote:
>>
>>> Must be a local environment thing, because AmpLab Jenkins can't
>>> reproduce it..... :-p
>>>
>>> On Wed, Oct 30, 2013 at 11:10 AM, Josh Rosen <rosenville@gmail.com> wrote:
>>> > Someone on the users list also encountered this exception:
>>> >
>>> >
>>> https://mail-archives.apache.org/mod_mbox/incubator-spark-user/201310.mbox/%3C64474308D680D540A4D8151B0F7C03F7025EF289%40SHSMSX104.ccr.corp.intel.com%3E
>>> >
>>> >
>>> > On Wed, Oct 30, 2013 at 9:40 AM, Evan Chan <ev@ooyala.com> wrote:
>>> >
>>> >> I'm at the latest
>>> >>
>>> >> commit f0e23a023ce1356bc0f04248605c48d4d08c2d05
>>> >> Merge: aec9bf9 a197137
>>> >> Author: Reynold Xin <rxin@apache.org>
>>> >> Date:   Tue Oct 29 01:41:44 2013 -0400
>>> >>
>>> >>
>>> >> and seeing this when I do a "test-only FileServerSuite":
>>> >>
>>> >> 13/10/30 09:35:04.300 INFO DAGScheduler: Completed ResultTask(0, 0)
>>> >> 13/10/30 09:35:04.307 INFO LocalTaskSetManager: Loss was due to
>>> >> java.io.StreamCorruptedException
>>> >> java.io.StreamCorruptedException: invalid type code: AC
>>> >>         at
>>> >> java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1353)
>>> >>         at
>>> java.io.ObjectInputStream.readObject(ObjectInputStream.java:348)
>>> >>         at
>>> >>
>>> org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:39)
>>> >>         at
>>> >>
>>> org.apache.spark.serializer.DeserializationStream$$anon$1.getNext(Serializer.scala:101)
>>> >>         at
>>> >> org.apache.spark.util.NextIterator.hasNext(NextIterator.scala:71)
>>> >>         at
>>> scala.collection.Iterator$$anon$21.hasNext(Iterator.scala:440)
>>> >>         at
>>> >>
>>> org.apache.spark.util.CompletionIterator.hasNext(CompletionIterator.scala:26)
>>> >>         at
>>> >>
>>> org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:27)
>>> >>         at
>>> >> org.apache.spark.Aggregator.combineCombinersByKey(Aggregator.scala:53)
>>> >>         at
>>> >>
>>> org.apache.spark.rdd.PairRDDFunctions$$anonfun$combineByKey$2.apply(PairRDDFunctions.scala:95)
>>> >>         at
>>> >>
>>> org.apache.spark.rdd.PairRDDFunctions$$anonfun$combineByKey$2.apply(PairRDDFunctions.scala:94)
>>> >>         at
>>> >>
>>> org.apache.spark.rdd.MapPartitionsWithContextRDD.compute(MapPartitionsWithContextRDD.scala:40)
>>> >>         at
>>> org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:237)
>>> >>         at org.apache.spark.rdd.RDD.iterator(RDD.scala:226)
>>> >>         at
>>> >> org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:107)
>>> >>         at org.apache.spark.scheduler.Task.run(Task.scala:53)
>>> >>         at
>>> >> org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:212)
>>> >>         at
>>> >>
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
>>> >>         at
>>> >>
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
>>> >>         at java.lang.Thread.run(Thread.java:680)
>>> >>
>>> >>
>>> >> Anybody else seen this yet?
>>> >>
>>> >> I have a really simple PR and this fails without my change, so I may
>>> >> go ahead and submit it anyways.
>>> >>
>>> >> --
>>> >> --
>>> >> Evan Chan
>>> >> Staff Engineer
>>> >> ev@ooyala.com  |
>>> >>
>>>
>>>
>>>
>>> --
>>> --
>>> Evan Chan
>>> Staff Engineer
>>> ev@ooyala.com  |
>>>
>
>
>
> --
> --
> Evan Chan
> Staff Engineer
> ev@ooyala.com  |



-- 
--
Evan Chan
Staff Engineer
ev@ooyala.com  |

Mime
View raw message