spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Eric Zhen <zhpeng...@gmail.com>
Subject Re: SparkSQL exception on spark.sql.codegen
Date Tue, 18 Nov 2014 05:37:06 GMT
Yes, it's always appears on a part of the whole tasks in a stage(i.e. 100/100
(65 failed)), and sometimes cause the stage to fail.

And there is another error that I'm not sure if there is a correlation.

java.lang.NoClassDefFoundError: Could not initialize class
org.apache.spark.sql.catalyst.expressions.codegen.GeneratePredicate$
        at
org.apache.spark.sql.execution.SparkPlan.newPredicate(SparkPlan.scala:114)
        at
org.apache.spark.sql.execution.Filter.conditionEvaluator$lzycompute(basicOperators.scala:55)
        at
org.apache.spark.sql.execution.Filter.conditionEvaluator(basicOperators.scala:55)
        at
org.apache.spark.sql.execution.Filter$$anonfun$2.apply(basicOperators.scala:58)
        at
org.apache.spark.sql.execution.Filter$$anonfun$2.apply(basicOperators.scala:57)
        at org.apache.spark.rdd.RDD$$anonfun$13.apply(RDD.scala:596)
        at org.apache.spark.rdd.RDD$$anonfun$13.apply(RDD.scala:596)
        at
org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
        at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:262)
        at org.apache.spark.rdd.RDD.iterator(RDD.scala:229)
        at
org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
        at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:262)
        at org.apache.spark.rdd.RDD.iterator(RDD.scala:229)
        at
org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
        at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:262)
        at org.apache.spark.rdd.RDD.iterator(RDD.scala:229)
        at
org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:68)
        at
org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:41)
        at org.apache.spark.scheduler.Task.run(Task.scala:54)
        at
org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:178)
        at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:744)

On Tue, Nov 18, 2014 at 11:41 AM, Michael Armbrust <michael@databricks.com>
wrote:

> Interesting, I believe we have run that query with version 1.1.0 with
> codegen turned on and not much has changed there.  Is the error
> deterministic?
>
> On Mon, Nov 17, 2014 at 7:04 PM, Eric Zhen <zhpeng.is@gmail.com> wrote:
>
>> Hi Michael,
>>
>> We use Spark v1.1.1-rc1 with jdk 1.7.0_51 and scala 2.10.4.
>>
>> On Tue, Nov 18, 2014 at 7:09 AM, Michael Armbrust <michael@databricks.com
>> > wrote:
>>
>>> What version of Spark SQL?
>>>
>>> On Sat, Nov 15, 2014 at 10:25 PM, Eric Zhen <zhpeng.is@gmail.com> wrote:
>>>
>>>> Hi all,
>>>>
>>>> We run SparkSQL on TPCDS benchmark Q19 with  spark.sql.codegen=true, we
>>>> got exceptions as below, has anyone else saw these before?
>>>>
>>>> java.lang.ExceptionInInitializerError
>>>>         at
>>>> org.apache.spark.sql.execution.SparkPlan.newProjection(SparkPlan.scala:92)
>>>>         at
>>>> org.apache.spark.sql.execution.Exchange$$anonfun$execute$1$$anonfun$1.apply(Exchange.scala:51)
>>>>         at
>>>> org.apache.spark.sql.execution.Exchange$$anonfun$execute$1$$anonfun$1.apply(Exchange.scala:48)
>>>>         at org.apache.spark.rdd.RDD$$anonfun$13.apply(RDD.scala:596)
>>>>         at org.apache.spark.rdd.RDD$$anonfun$13.apply(RDD.scala:596)
>>>>         at
>>>> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
>>>>         at
>>>> org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:262)
>>>>         at org.apache.spark.rdd.RDD.iterator(RDD.scala:229)
>>>>         at
>>>> org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:68)
>>>>         at
>>>> org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:41)
>>>>         at org.apache.spark.scheduler.Task.run(Task.scala:54)
>>>>         at
>>>> org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:178)
>>>>         at
>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>>>>         at
>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>>>>         at java.lang.Thread.run(Thread.java:744)
>>>> Caused by: java.lang.NullPointerException
>>>>         at
>>>> scala.reflect.internal.Types$TypeRef.computeHashCode(Types.scala:2358)
>>>>         at
>>>> scala.reflect.internal.Types$UniqueType.<init>(Types.scala:1304)
>>>>         at scala.reflect.internal.Types$TypeRef.<init>(Types.scala:2341)
>>>>         at
>>>> scala.reflect.internal.Types$NoArgsTypeRef.<init>(Types.scala:2137)
>>>>         at
>>>> scala.reflect.internal.Types$TypeRef$$anon$6.<init>(Types.scala:2544)
>>>>         at scala.reflect.internal.Types$TypeRef$.apply(Types.scala:2544)
>>>>         at scala.reflect.internal.Types$class.typeRef(Types.scala:3615)
>>>>         at
>>>> scala.reflect.internal.SymbolTable.typeRef(SymbolTable.scala:13)
>>>>         at
>>>> scala.reflect.internal.Symbols$TypeSymbol.newTypeRef(Symbols.scala:2752)
>>>>         at
>>>> scala.reflect.internal.Symbols$TypeSymbol.typeConstructor(Symbols.scala:2806)
>>>>         at
>>>> scala.reflect.internal.Symbols$SymbolContextApiImpl.toTypeConstructor(Symbols.scala:103)
>>>>         at
>>>> scala.reflect.internal.Symbols$TypeSymbol.toTypeConstructor(Symbols.scala:2698)
>>>>         at
>>>> org.apache.spark.sql.catalyst.expressions.codegen.CodeGenerator$$typecreator1$1.apply(CodeGenerator.scala:46)
>>>>         at
>>>> scala.reflect.api.TypeTags$WeakTypeTagImpl.tpe$lzycompute(TypeTags.scala:231)
>>>>         at
>>>> scala.reflect.api.TypeTags$WeakTypeTagImpl.tpe(TypeTags.scala:231)
>>>>         at scala.reflect.api.TypeTags$class.typeOf(TypeTags.scala:335)
>>>>         at scala.reflect.api.Universe.typeOf(Universe.scala:59)
>>>>         at
>>>> org.apache.spark.sql.catalyst.expressions.codegen.CodeGenerator.<init>(CodeGenerator.scala:46)
>>>>         at
>>>> org.apache.spark.sql.catalyst.expressions.codegen.GenerateProjection$.<init>(GenerateProjection.scala:29)
>>>>         at
>>>> org.apache.spark.sql.catalyst.expressions.codegen.GenerateProjection$.<clinit>(GenerateProjection.scala)
>>>>         ... 15 more
>>>> --
>>>> Best Regards
>>>>
>>>
>>>
>>
>>
>> --
>> Best Regards
>>
>
>


-- 
Best Regards

Mime
View raw message