spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Michael Armbrust <mich...@databricks.com>
Subject Re: SparkSQL exception on spark.sql.codegen
Date Tue, 18 Nov 2014 03:41:13 GMT
Interesting, I believe we have run that query with version 1.1.0 with
codegen turned on and not much has changed there.  Is the error
deterministic?

On Mon, Nov 17, 2014 at 7:04 PM, Eric Zhen <zhpeng.is@gmail.com> wrote:

> Hi Michael,
>
> We use Spark v1.1.1-rc1 with jdk 1.7.0_51 and scala 2.10.4.
>
> On Tue, Nov 18, 2014 at 7:09 AM, Michael Armbrust <michael@databricks.com>
> wrote:
>
>> What version of Spark SQL?
>>
>> On Sat, Nov 15, 2014 at 10:25 PM, Eric Zhen <zhpeng.is@gmail.com> wrote:
>>
>>> Hi all,
>>>
>>> We run SparkSQL on TPCDS benchmark Q19 with  spark.sql.codegen=true, we
>>> got exceptions as below, has anyone else saw these before?
>>>
>>> java.lang.ExceptionInInitializerError
>>>         at
>>> org.apache.spark.sql.execution.SparkPlan.newProjection(SparkPlan.scala:92)
>>>         at
>>> org.apache.spark.sql.execution.Exchange$$anonfun$execute$1$$anonfun$1.apply(Exchange.scala:51)
>>>         at
>>> org.apache.spark.sql.execution.Exchange$$anonfun$execute$1$$anonfun$1.apply(Exchange.scala:48)
>>>         at org.apache.spark.rdd.RDD$$anonfun$13.apply(RDD.scala:596)
>>>         at org.apache.spark.rdd.RDD$$anonfun$13.apply(RDD.scala:596)
>>>         at
>>> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
>>>         at
>>> org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:262)
>>>         at org.apache.spark.rdd.RDD.iterator(RDD.scala:229)
>>>         at
>>> org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:68)
>>>         at
>>> org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:41)
>>>         at org.apache.spark.scheduler.Task.run(Task.scala:54)
>>>         at
>>> org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:178)
>>>         at
>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>>>         at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>>>         at java.lang.Thread.run(Thread.java:744)
>>> Caused by: java.lang.NullPointerException
>>>         at
>>> scala.reflect.internal.Types$TypeRef.computeHashCode(Types.scala:2358)
>>>         at
>>> scala.reflect.internal.Types$UniqueType.<init>(Types.scala:1304)
>>>         at scala.reflect.internal.Types$TypeRef.<init>(Types.scala:2341)
>>>         at
>>> scala.reflect.internal.Types$NoArgsTypeRef.<init>(Types.scala:2137)
>>>         at
>>> scala.reflect.internal.Types$TypeRef$$anon$6.<init>(Types.scala:2544)
>>>         at scala.reflect.internal.Types$TypeRef$.apply(Types.scala:2544)
>>>         at scala.reflect.internal.Types$class.typeRef(Types.scala:3615)
>>>         at
>>> scala.reflect.internal.SymbolTable.typeRef(SymbolTable.scala:13)
>>>         at
>>> scala.reflect.internal.Symbols$TypeSymbol.newTypeRef(Symbols.scala:2752)
>>>         at
>>> scala.reflect.internal.Symbols$TypeSymbol.typeConstructor(Symbols.scala:2806)
>>>         at
>>> scala.reflect.internal.Symbols$SymbolContextApiImpl.toTypeConstructor(Symbols.scala:103)
>>>         at
>>> scala.reflect.internal.Symbols$TypeSymbol.toTypeConstructor(Symbols.scala:2698)
>>>         at
>>> org.apache.spark.sql.catalyst.expressions.codegen.CodeGenerator$$typecreator1$1.apply(CodeGenerator.scala:46)
>>>         at
>>> scala.reflect.api.TypeTags$WeakTypeTagImpl.tpe$lzycompute(TypeTags.scala:231)
>>>         at
>>> scala.reflect.api.TypeTags$WeakTypeTagImpl.tpe(TypeTags.scala:231)
>>>         at scala.reflect.api.TypeTags$class.typeOf(TypeTags.scala:335)
>>>         at scala.reflect.api.Universe.typeOf(Universe.scala:59)
>>>         at
>>> org.apache.spark.sql.catalyst.expressions.codegen.CodeGenerator.<init>(CodeGenerator.scala:46)
>>>         at
>>> org.apache.spark.sql.catalyst.expressions.codegen.GenerateProjection$.<init>(GenerateProjection.scala:29)
>>>         at
>>> org.apache.spark.sql.catalyst.expressions.codegen.GenerateProjection$.<clinit>(GenerateProjection.scala)
>>>         ... 15 more
>>> --
>>> Best Regards
>>>
>>
>>
>
>
> --
> Best Regards
>

Mime
View raw message