spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jacek Laskowski <ja...@japila.pl>
Subject Re: Strange ML pipeline errors from HashingTF using v1.6.1
Date Tue, 29 Mar 2016 05:45:28 GMT
Hi,

How do you run the pipeline? Do you assembly or package? Is this on
local or spark or other cluster manager? What's the build
configuration?

Pozdrawiam,
Jacek Laskowski
----
https://medium.com/@jaceklaskowski/
Mastering Apache Spark http://bit.ly/mastering-apache-spark
Follow me at https://twitter.com/jaceklaskowski


On Mon, Mar 28, 2016 at 7:11 PM, Timothy Potter <thelabdude@gmail.com> wrote:
> I'm seeing the following error when trying to generate a prediction
> from a very simple ML pipeline based model. I've verified that the raw
> data sent to the tokenizer is valid (not null). It seems like this is
> some sort of weird classpath or class loading type issue. Any help you
> can provide in trying to troubleshoot this further would be
> appreciated.
>
>  Error in machine-learning, docId=20news-18828/alt.atheism/51176
> scala.reflect.internal.Symbols$CyclicReference: illegal cyclic
> reference involving package <root>
>     at scala.reflect.internal.Symbols$TypeSymbol.tpe(Symbols.scala:2768)
> ~[scala-reflect-2.10.5.jar:?]
>     at scala.reflect.internal.Mirrors$Roots$RootPackage$.<init>(Mirrors.scala:268)
> ~[scala-reflect-2.10.5.jar:?]
>     at scala.reflect.internal.Mirrors$Roots.RootPackage$lzycompute(Mirrors.scala:267)
> ~[scala-reflect-2.10.5.jar:?]
>     at scala.reflect.internal.Mirrors$Roots.RootPackage(Mirrors.scala:267)
> ~[scala-reflect-2.10.5.jar:?]
>     at scala.reflect.runtime.JavaMirrors$JavaMirror.scala$reflect$runtime$JavaMirrors$$makeScalaPackage(JavaMirrors.scala:902)
> ~[scala-reflect-2.10.5.jar:?]
>     at scala.reflect.runtime.JavaMirrors$class.missingHook(JavaMirrors.scala:1299)
> ~[scala-reflect-2.10.5.jar:?]
>     at scala.reflect.runtime.JavaUniverse.missingHook(JavaUniverse.scala:12)
> ~[scala-reflect-2.10.5.jar:?]
>     at scala.reflect.internal.Mirrors$RootsBase.universeMissingHook(Mirrors.scala:77)
> ~[scala-reflect-2.10.5.jar:?]
>     at scala.reflect.internal.Mirrors$RootsBase.missingHook(Mirrors.scala:79)
> ~[scala-reflect-2.10.5.jar:?]
>     at scala.reflect.internal.Mirrors$RootsBase.getModuleOrClass(Mirrors.scala:48)
> ~[scala-reflect-2.10.5.jar:?]
>     at scala.reflect.internal.Mirrors$RootsBase.getModuleOrClass(Mirrors.scala:40)
> ~[scala-reflect-2.10.5.jar:?]
>     at scala.reflect.internal.Mirrors$RootsBase.getModuleOrClass(Mirrors.scala:40)
> ~[scala-reflect-2.10.5.jar:?]
>     at scala.reflect.internal.Mirrors$RootsBase.getModuleOrClass(Mirrors.scala:40)
> ~[scala-reflect-2.10.5.jar:?]
>     at scala.reflect.internal.Mirrors$RootsBase.getModuleOrClass(Mirrors.scala:40)
> ~[scala-reflect-2.10.5.jar:?]
>     at scala.reflect.internal.Mirrors$RootsBase.getModuleOrClass(Mirrors.scala:40)
> ~[scala-reflect-2.10.5.jar:?]
>     at scala.reflect.internal.Mirrors$RootsBase.getModuleOrClass(Mirrors.scala:61)
> ~[scala-reflect-2.10.5.jar:?]
>     at scala.reflect.internal.Mirrors$RootsBase.staticModuleOrClass(Mirrors.scala:72)
> ~[scala-reflect-2.10.5.jar:?]
>     at scala.reflect.internal.Mirrors$RootsBase.staticClass(Mirrors.scala:119)
> ~[scala-reflect-2.10.5.jar:?]
>     at scala.reflect.internal.Mirrors$RootsBase.staticClass(Mirrors.scala:21)
> ~[scala-reflect-2.10.5.jar:?]
>     at org.apache.spark.ml.feature.HashingTF$$typecreator1$1.apply(HashingTF.scala:66)
> ~[spark-mllib_2.10-1.6.1.jar:1.6.1]
>     at scala.reflect.api.TypeTags$WeakTypeTagImpl.tpe$lzycompute(TypeTags.scala:231)
> ~[scala-reflect-2.10.5.jar:?]
>     at scala.reflect.api.TypeTags$WeakTypeTagImpl.tpe(TypeTags.scala:231)
> ~[scala-reflect-2.10.5.jar:?]
>     at org.apache.spark.sql.catalyst.ScalaReflection$class.localTypeOf(ScalaReflection.scala:654)
> ~[spark-catalyst_2.10-1.6.1.jar:1.6.1]
>     at org.apache.spark.sql.catalyst.ScalaReflection$.localTypeOf(ScalaReflection.scala:30)
> ~[spark-catalyst_2.10-1.6.1.jar:1.6.1]
>     at org.apache.spark.sql.catalyst.ScalaReflection$class.schemaFor(ScalaReflection.scala:642)
> ~[spark-catalyst_2.10-1.6.1.jar:1.6.1]
>     at org.apache.spark.sql.catalyst.ScalaReflection$.schemaFor(ScalaReflection.scala:30)
> ~[spark-catalyst_2.10-1.6.1.jar:1.6.1]
>     at org.apache.spark.sql.functions$.udf(functions.scala:2576)
> ~[spark-sql_2.10-1.6.1.jar:1.6.1]
>     at org.apache.spark.ml.feature.HashingTF.transform(HashingTF.scala:66)
> ~[spark-mllib_2.10-1.6.1.jar:1.6.1]
>     at org.apache.spark.ml.PipelineModel$$anonfun$transform$1.apply(Pipeline.scala:297)
> ~[spark-mllib_2.10-1.6.1.jar:1.6.1]
>     at org.apache.spark.ml.PipelineModel$$anonfun$transform$1.apply(Pipeline.scala:297)
> ~[spark-mllib_2.10-1.6.1.jar:1.6.1]
>     at scala.collection.IndexedSeqOptimized$class.foldl(IndexedSeqOptimized.scala:51)
> ~[scala-library-2.10.5.jar:?]
>     at scala.collection.IndexedSeqOptimized$class.foldLeft(IndexedSeqOptimized.scala:60)
> ~[scala-library-2.10.5.jar:?]
>     at scala.collection.mutable.ArrayOps$ofRef.foldLeft(ArrayOps.scala:108)
> ~[scala-library-2.10.5.jar:?]
>     at org.apache.spark.ml.PipelineModel.transform(Pipeline.scala:297)
> ~[spark-mllib_2.10-1.6.1.jar:1.6.1]
>     at org.apache.spark.ml.tuning.CrossValidatorModel.transform(CrossValidator.scala:338)
> ~[spark-mllib_2.10-1.6.1.jar:1.6.1]
>
>
> I've also seen similar errors such as:
>
> java.lang.AssertionError: assertion failed: List(package linalg, package linalg)
>     at scala.reflect.internal.Symbols$Symbol.suchThat(Symbols.scala:1678)
> ~[scala-reflect-2.10.5.jar:?]
>     at scala.reflect.internal.Mirrors$RootsBase.getModuleOrClass(Mirrors.scala:44)
> ~[scala-reflect-2.10.5.jar:?]
>     at scala.reflect.internal.Mirrors$RootsBase.getModuleOrClass(Mirrors.scala:40)
> ~[scala-reflect-2.10.5.jar:?]
>     at scala.reflect.internal.Mirrors$RootsBase.getModuleOrClass(Mirrors.scala:61)
> ~[scala-reflect-2.10.5.jar:?]
>     at scala.reflect.internal.Mirrors$RootsBase.staticModuleOrClass(Mirrors.scala:72)
> ~[scala-reflect-2.10.5.jar:?]
>     at scala.reflect.internal.Mirrors$RootsBase.staticClass(Mirrors.scala:119)
> ~[scala-reflect-2.10.5.jar:?]
>     at scala.reflect.internal.Mirrors$RootsBase.staticClass(Mirrors.scala:21)
> ~[scala-reflect-2.10.5.jar:?]
>     at org.apache.spark.ml.feature.HashingTF$$typecreator1$1.apply(HashingTF.scala:66)
> ~[spark-mllib_2.10-1.6.1.jar:1.6.1]
>     at scala.reflect.api.TypeTags$WeakTypeTagImpl.tpe$lzycompute(TypeTags.scala:231)
> ~[scala-reflect-2.10.5.jar:?]
>     at scala.reflect.api.TypeTags$WeakTypeTagImpl.tpe(TypeTags.scala:231)
> ~[scala-reflect-2.10.5.jar:?]
>     at org.apache.spark.sql.catalyst.ScalaReflection$class.localTypeOf(ScalaReflection.scala:654)
> ~[spark-catalyst_2.10-1.6.1.jar:1.6.1]
>     at org.apache.spark.sql.catalyst.ScalaReflection$.localTypeOf(ScalaReflection.scala:30)
> ~[spark-catalyst_2.10-1.6.1.jar:1.6.1]
>     at org.apache.spark.sql.catalyst.ScalaReflection$class.schemaFor(ScalaReflection.scala:642)
> ~[spark-catalyst_2.10-1.6.1.jar:1.6.1]
>     at org.apache.spark.sql.catalyst.ScalaReflection$.schemaFor(ScalaReflection.scala:30)
> ~[spark-catalyst_2.10-1.6.1.jar:1.6.1]
>     at org.apache.spark.sql.functions$.udf(functions.scala:2576)
> ~[spark-sql_2.10-1.6.1.jar:1.6.1]
>     at org.apache.spark.ml.feature.HashingTF.transform(HashingTF.scala:66)
> ~[spark-mllib_2.10-1.6.1.jar:1.6.1]
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
>

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org


Mime
View raw message