spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From lalit1303 <la...@sigmoidanalytics.com>
Subject Re: java.lang.StackOverflowError when calling count()
Date Wed, 14 May 2014 10:32:36 GMT
If we do cache() + count() after say every 50 iterations. The whole process
becomes very slow.
I have tried checkpoint() , cache() + count(), saveAsObjectFiles().
Nothing works.
Materializing RDD's lead to drastic decrease in performance & if we don't
materialize, we face stackoverflowerror.


On Wed, May 14, 2014 at 10:25 AM, Nick Chammas [via Apache Spark User List]
<ml-node+s1001560n5683h45@n3.nabble.com> wrote:

> Would cache() + count() every N iterations work just as well as
> checkPoint() + count() to get around this issue?
>
> We're basically trying to get Spark to avoid working on too lengthy a
> lineage at once, right?
>
> Nick
>
>
> On Tue, May 13, 2014 at 12:04 PM, Xiangrui Meng <[hidden email]<http://user/SendEmail.jtp?type=node&node=5683&i=0>
> > wrote:
>
>> After checkPoint, call count directly to materialize it. -Xiangrui
>>
>> On Tue, May 13, 2014 at 4:20 AM, Mayur Rustagi <[hidden email]<http://user/SendEmail.jtp?type=node&node=5683&i=1>>
>> wrote:
>> > We are running into same issue. After 700 or so files the stack
>> overflows,
>> > cache, persist & checkpointing dont help.
>> > Basically checkpointing only saves the RDD when it is materialized & it
>> only
>> > materializes in the end, then it runs out of stack.
>> >
>> > Regards
>> > Mayur
>> >
>> > Mayur Rustagi
>> > Ph: <a href="tel:%2B1%20%28760%29%20203%203257" value="+17602033257">+1
>> (760) 203 3257
>> > http://www.sigmoidanalytics.com
>> > @mayur_rustagi
>>
>> >
>> >
>> >
>> > On Tue, May 13, 2014 at 11:40 AM, Xiangrui Meng <[hidden email]<http://user/SendEmail.jtp?type=node&node=5683&i=2>>
>> wrote:
>> >>
>> >> You have a long lineage that causes the StackOverflow error. Try
>> >> rdd.checkPoint() and rdd.count() for every 20~30 iterations.
>> >> checkPoint can cut the lineage. -Xiangrui
>> >>
>> >> On Mon, May 12, 2014 at 3:42 PM, Guanhua Yan <[hidden email]<http://user/SendEmail.jtp?type=node&node=5683&i=3>>
>> wrote:
>> >> > Dear Sparkers:
>> >> >
>> >> > I am using Python spark of version 0.9.0 to implement some iterative
>> >> > algorithm. I got some errors shown at the end of this email. It seems
>> >> > that
>> >> > it's due to the Java Stack Overflow error. The same error has been
>> >> > duplicated on a mac desktop and a linux workstation, both running the
>> >> > same
>> >> > version of Spark.
>> >> >
>> >> > The same line of code works correctly after quite some iterations.
At
>> >> > the
>> >> > line of error, rdd__new.count() could be 0. (In some previous rounds,
>> >> > this
>> >> > was also 0 without any problem).
>> >> >
>> >> > Any thoughts on this?
>> >> >
>> >> > Thank you very much,
>> >> > - Guanhua
>> >> >
>> >> >
>> >> > ========================================
>> >> > CODE:    print "round", round, rdd__new.count()
>> >> > ========================================
>> >> >   File
>> >> >
>> >> >
>> "/home1/ghyan/Software/spark-0.9.0-incubating-bin-hadoop2/python/pyspark/rdd.py",
>> >> > line 542, in count
>> >> > 14/05/12 16:20:28 INFO TaskSetManager: Loss was due to
>> >> > java.lang.StackOverflowError [duplicate 1]
>> >> >     return self.mapPartitions(lambda i: [sum(1 for _ in i)]).sum()
>> >> > 14/05/12 16:20:28 ERROR TaskSetManager: Task 8419.0:0 failed 1 times;
>> >> > aborting job
>> >> >   File
>> >> >
>> >> >
>> "/home1/ghyan/Software/spark-0.9.0-incubating-bin-hadoop2/python/pyspark/rdd.py",
>> >> > line 533, in sum
>> >> > 14/05/12 16:20:28 INFO TaskSchedulerImpl: Ignoring update with state
>> >> > FAILED
>> >> > from TID 1774 because its task set is gone
>> >> >     return self.mapPartitions(lambda x:
>> [sum(x)]).reduce(operator.add)
>> >> >   File
>> >> >
>> >> >
>> "/home1/ghyan/Software/spark-0.9.0-incubating-bin-hadoop2/python/pyspark/rdd.py",
>> >> > line 499, in reduce
>> >> >     vals = self.mapPartitions(func).collect()
>> >> >   File
>> >> >
>> >> >
>> "/home1/ghyan/Software/spark-0.9.0-incubating-bin-hadoop2/python/pyspark/rdd.py",
>> >> > line 463, in collect
>> >> >     bytesInJava = self._jrdd.collect().iterator()
>> >> >   File
>> >> >
>> >> >
>> "/home1/ghyan/Software/spark-0.9.0-incubating-bin-hadoop2/python/lib/py4j-0.8.1-src.zip/py4j/java_gateway.py",
>> >> > line 537, in __call__
>> >> >   File
>> >> >
>> >> >
>> "/home1/ghyan/Software/spark-0.9.0-incubating-bin-hadoop2/python/lib/py4j-0.8.1-src.zip/py4j/protocol.py",
>> >> > line 300, in get_return_value
>> >> > py4j.protocol.Py4JJavaError: An error occurred while calling
>> >> > o4317.collect.
>> >> > : org.apache.spark.SparkException: Job aborted: Task 8419.0:1 failed
>> 1
>> >> > times
>> >> > (most recent failure: Exception failure:
>> java.lang.StackOverflowError)
>> >> > at
>> >> >
>> >> >
>> org.apache.spark.scheduler.DAGScheduler$$anonfun$org$apache$spark$scheduler$DAGScheduler$$abortStage$1.apply(DAGScheduler.scala:1028)
>> >> > at
>> >> >
>> >> >
>> org.apache.spark.scheduler.DAGScheduler$$anonfun$org$apache$spark$scheduler$DAGScheduler$$abortStage$1.apply(DAGScheduler.scala:1026)
>> >> > at
>> >> >
>> >> >
>> scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
>> >> > at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47)
>> >> > at
>> >> >
>> >> > org.apache.spark.scheduler.DAGScheduler.org
>> $apache$spark$scheduler$DAGScheduler$$abortStage(DAGScheduler.scala:1026)
>> >> > at
>> >> >
>> >> >
>> org.apache.spark.scheduler.DAGScheduler$$anonfun$processEvent$10.apply(DAGScheduler.scala:619)
>> >> > at
>> >> >
>> >> >
>> org.apache.spark.scheduler.DAGScheduler$$anonfun$processEvent$10.apply(DAGScheduler.scala:619)
>> >> > at scala.Option.foreach(Option.scala:236)
>> >> > at
>> >> >
>> >> >
>> org.apache.spark.scheduler.DAGScheduler.processEvent(DAGScheduler.scala:619)
>> >> > at
>> >> >
>> >> >
>> org.apache.spark.scheduler.DAGScheduler$$anonfun$start$1$$anon$2$$anonfun$receive$1.applyOrElse(DAGScheduler.scala:207)
>> >> > at akka.actor.ActorCell.receiveMessage(ActorCell.scala:498)
>> >> > at akka.actor.ActorCell.invoke(ActorCell.scala:456)
>> >> > at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:237)
>> >> > at akka.dispatch.Mailbox.run(Mailbox.scala:219)
>> >> > at
>> >> >
>> >> >
>> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:386)
>> >> > at
>> scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
>> >> > at
>> >> >
>> >> >
>> scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
>> >> > at
>> >> >
>> scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
>> >> > at
>> >> >
>> >> >
>> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>> >> >
>> >> > ======================================
>> >> > The stack overflow error is shown as follows:
>> >> > ======================================
>> >> >
>> >> > 14/05/12 16:20:28 ERROR Executor: Exception in task ID 1774
>> >> > java.lang.StackOverflowError
>> >> > at java.util.zip.Inflater.inflate(Inflater.java:259)
>> >> > at
>> java.util.zip.InflaterInputStream.read(InflaterInputStream.java:152)
>> >> > at java.util.zip.GZIPInputStream.read(GZIPInputStream.java:116)
>> >> > at
>> >> >
>> >> >
>> java.io.ObjectInputStream$PeekInputStream.read(ObjectInputStream.java:2310)
>> >> > at
>> >> >
>> >> >
>> java.io.ObjectInputStream$PeekInputStream.readFully(ObjectInputStream.java:2323)
>> >> > at
>> >> >
>> >> >
>> java.io.ObjectInputStream$BlockDataInputStream.readInt(ObjectInputStream.java:2818)
>> >> > at java.io.ObjectInputStream.readHandle(ObjectInputStream.java:1452)
>> >> > at
>> java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1511)
>> >> > at
>> >> >
>> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1771)
>> >> > at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
>> >> > at
>> >> >
>> java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1990)
>> >> > at
>> java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1915)
>> >> > at
>> >> >
>> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798)
>> >> > at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
>> >> > at java.io.ObjectInputStream.readArray(ObjectInputStream.java:1706)
>> >> > at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1344)
>> >> > at
>> >> >
>> java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1990)
>> >> > at
>> java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1915)
>> >> > at
>> >> >
>> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798)
>> >> > at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
>> >> > at
>> >> >
>> java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1990)
>> >> > at
>> java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1915)
>> >> > at
>> >> >
>> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798)
>> >> > at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
>> >> > at
>> >> >
>> java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1990)
>> >> > at
>> java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1915)
>> >> > at
>> >> >
>> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798)
>> >> > at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
>> >> > at java.io.ObjectInputStream.readObject(ObjectInputStream.java:370)
>> >> > at scala.collection.immutable.$colon$colon.readObject(List.scala:362)
>> >> > at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source)
>> >> > at
>> >> >
>> >> >
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>> >> > at java.lang.reflect.Method.invoke(Method.java:606)
>> >> > at
>> >> >
>> java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1017)
>> >> > at
>> java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1893)
>> >> > at
>> >> >
>> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798)
>> >> > at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
>> >> > at
>> >> >
>> java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1990)
>> >> > at
>> java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1915)
>> >> > at
>> >> >
>> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798)
>> >> > at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
>> >> > at
>> >> >
>> java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1990)
>> >> > at
>> java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1915)
>> >> > at
>> >> >
>> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798)
>> >> > at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
>> >> > at java.io.ObjectInputStream.readObject(ObjectInputStream.java:370)
>> >> > at scala.collection.immutable.$colon$colon.readObject(List.scala:362)
>> >> > at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source)
>> >> > at
>> >> >
>> >> >
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>> >> > at java.lang.reflect.Method.invoke(Method.java:606)
>> >> > at
>> >> >
>> java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1017)
>> >> > at
>> java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1893)
>> >> > at
>> >> >
>> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798)
>> >> > at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
>> >> > at
>> >> >
>> java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1990)
>> >> > at
>> java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1915)
>> >> > at
>> >> >
>> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798)
>> >> > at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
>> >> > at
>> >> >
>> java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1990)
>> >> > at
>> java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1915)
>> >> > at
>> >> >
>> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798)
>> >> > at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
>> >> > at java.io.ObjectInputStream.readObject(ObjectInputStream.java:370)
>> >> > at scala.collection.immutable.$colon$colon.readObject(List.scala:362)
>> >> > at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source)
>> >> >  The above replicated many times after this …
>> >> > ======================================
>> >
>> >
>>
>
>
>
> ------------------------------
>  If you reply to this email, your message will be added to the discussion
> below:
>
> http://apache-spark-user-list.1001560.n3.nabble.com/java-lang-StackOverflowError-when-calling-count-tp5649p5683.html
>  To start a new topic under Apache Spark User List, email
> ml-node+s1001560n1h3@n3.nabble.com
> To unsubscribe from Apache Spark User List, click here<http://apache-spark-user-list.1001560.n3.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=1&code=bGFsaXRAc2lnbW9pZGFuYWx5dGljcy5jb218MXwtMTIwMzUwMjA2MQ==>
> .
> NAML<http://apache-spark-user-list.1001560.n3.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>



-- 
-- 
Thanks & Regards,
Lalit Yadav
+91-9901007692




-----
Lalit Yadav
lalit@sigmoidanalytics.com
--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/java-lang-StackOverflowError-when-calling-count-tp5649p5698.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
Mime
View raw message