spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "FangFang Chen" <lulynn_2015_sp...@163.com>
Subject Spark sql with large sql syntax job failed with outofmemory error and grows beyond 64k warn
Date Mon, 25 Apr 2016 04:49:48 GMT
Hi all,
With large sql command, job failed with following error. Please give your suggestion on how
to resolve it. Thanks


Sql file size: 676k
Log:
16/04/25 10:55:00 WARN TaskSetManager: Lost task 84.0 in stage 0.0 (TID 6, BJHC-HADOOP-HERA-17493.jd.local):
java.util.concurrent.ExecutionException: java.lang.Exception: failed to compile: org.codehaus.janino.JaninoRuntimeException:
Code of method "(Lorg/apache/spark/sql/catalyst/expressions/GeneratedClass$SpecificUnsafeProjection;Lorg/apache/spark/sql/catalyst/InternalRow;)V"
of class "org.apache.spark.sql.catalyst.expressions.GeneratedClass$SpecificUnsafeProjection"
grows beyond 64 KB


public Object generate(org.apache.spark.sql.catalyst.expressions.Expression[] exprs) {
  return new SpecificUnsafeProjection(exprs);
}


class SpecificUnsafeProjection extends org.apache.spark.sql.catalyst.expressions.UnsafeProjection
{




......


java.lang.OutOfMemoryError: Java heap space
  at com.google.protobuf.ByteString.copyFrom(ByteString.java:192)
  at com.google.protobuf.CodedInputStream.readBytes(CodedInputStream.java:324)
  at akka.remote.WireFormats$AkkaProtocolMessage.<init>(WireFormats.java:6657)
  at akka.remote.WireFormats$AkkaProtocolMessage.<init>(WireFormats.java:6607)
  at akka.remote.WireFormats$AkkaProtocolMessage$1.parsePartialFrom(WireFormats.java:6703)
  at akka.remote.WireFormats$AkkaProtocolMessage$1.parsePartialFrom(WireFormats.java:6698)
  at com.google.protobuf.AbstractParser.parsePartialFrom(AbstractParser.java:141)
  at com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:176)
  at com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:188)
  at com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:193)
  at com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:49)
  at akka.remote.WireFormats$AkkaProtocolMessage.parseFrom(WireFormats.java:6821)
  at akka.remote.transport.AkkaPduProtobufCodec$.decodePdu(AkkaPduCodec.scala:168)


  


发自 网易邮箱大师
Mime
View raw message