spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Zhan Zhang <zzh...@hortonworks.com>
Subject Re: Driver hangs on running mllib word2vec
Date Tue, 06 Jan 2015 06:10:54 GMT
I think it is overflow. The training data is quite big. The algorithms  scalability highly
depends on the vocabSize. Even without overflow, there are still other bottlenecks, for example,
syn0Global and syn1Global, each of them has vocabSize * vectorSize elements.

Thanks.

Zhan Zhang


On Jan 5, 2015, at 7:47 PM, Eric Zhen <zhpeng.is@gmail.com> wrote:

> Hi Xiangrui,
> 
> Our dataset is about 80GB(10B lines). 
> 
> In the driver's log, we foud this:
> 
> INFO Word2Vec: trainWordsCount = -1610413239
> 
> it seems that there is a integer overflow?
> 
> 
> On Tue, Jan 6, 2015 at 5:44 AM, Xiangrui Meng <mengxr@gmail.com> wrote:
> How big is your dataset, and what is the vocabulary size? -Xiangrui
> 
> On Sun, Jan 4, 2015 at 11:18 PM, Eric Zhen <zhpeng.is@gmail.com> wrote:
> > Hi,
> >
> > When we run mllib word2vec(spark-1.1.0), driver get stuck with 100% cup
> > usage. Here is the jstack output:
> >
> > "main" prio=10 tid=0x0000000040112800 nid=0x46f2 runnable
> > [0x000000004162e000]
> >    java.lang.Thread.State: RUNNABLE
> >         at
> > java.io.ObjectOutputStream$BlockDataOutputStream.drain(ObjectOutputStream.java:1847)
> >         at
> > java.io.ObjectOutputStream$BlockDataOutputStream.write(ObjectOutputStream.java:1778)
> >         at java.io.DataOutputStream.writeInt(DataOutputStream.java:182)
> >         at java.io.DataOutputStream.writeFloat(DataOutputStream.java:225)
> >         at
> > java.io.ObjectOutputStream$BlockDataOutputStream.writeFloats(ObjectOutputStream.java:2064)
> >         at
> > java.io.ObjectOutputStream.writeArray(ObjectOutputStream.java:1310)
> >         at
> > java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1154)
> >         at
> > java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1518)
> >         at
> > java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1483)
> >         at
> > java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1400)
> >         at
> > java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1158)
> >         at
> > java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1518)
> >         at
> > java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1483)
> >         at
> > java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1400)
> >         at
> > java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1158)
> >         at
> > java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1518)
> >         at
> > java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1483)
> >         at
> > java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1400)
> >         at
> > java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1158)
> >         at
> > java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:330)
> >         at
> > org.apache.spark.serializer.JavaSerializationStream.writeObject(JavaSerializer.scala:42)
> >         at
> > org.apache.spark.serializer.JavaSerializerInstance.serialize(JavaSerializer.scala:73)
> >         at
> > org.apache.spark.util.ClosureCleaner$.ensureSerializable(ClosureCleaner.scala:164)
> >         at
> > org.apache.spark.util.ClosureCleaner$.clean(ClosureCleaner.scala:158)
> >         at org.apache.spark.SparkContext.clean(SparkContext.scala:1242)
> >         at org.apache.spark.rdd.RDD.mapPartitionsWithIndex(RDD.scala:610)
> >         at
> > org.apache.spark.mllib.feature.Word2Vec$$anonfun$fit$1.apply$mcVI$sp(Word2Vec.scala:291)
> >         at scala.collection.immutable.Range.foreach$mVc$sp(Range.scala:141)
> >         at org.apache.spark.mllib.feature.Word2Vec.fit(Word2Vec.scala:290)
> >         at com.baidu.inf.WordCount$.main(WordCount.scala:31)
> >         at com.baidu.inf.WordCount.main(WordCount.scala)
> >         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> >         at
> > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> >         at
> > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> >         at java.lang.reflect.Method.invoke(Method.java:597)
> >         at
> > org.apache.spark.deploy.SparkSubmit$.launch(SparkSubmit.scala:328)
> >         at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:75)
> >         at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
> >
> > --
> > Best Regards
> 
> 
> 
> -- 
> Best Regards


-- 
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to 
which it is addressed and may contain information that is confidential, 
privileged and exempt from disclosure under applicable law. If the reader 
of this message is not the intended recipient, you are hereby notified that 
any printing, copying, dissemination, distribution, disclosure or 
forwarding of this communication is strictly prohibited. If you have 
received this communication in error, please contact the sender immediately 
and delete it from your system. Thank You.

Mime
View raw message