spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Nimmi Cv <nimm...@gmail.com>
Subject org.apache.spark.util.SparkUncaughtExceptionHandler
Date Thu, 10 Oct 2019 20:39:22 GMT
0

I get the following error on executors while running my spark job. I am
reading data from Database. The data has string in UTF8

Iterator t.next().getString(row.fieldIndex("short_name"));

ERROR org.apache.spark.util.SparkUncaughtExceptionHandler - Uncaught
exception in thread Thread[Executor task launch worker for task 359,5,main]
java.lang.OutOfMemoryError: Java heap space at
org.apache.spark.unsafe.types.UTF8String.fromAddress(UTF8String.java:135)
at
org.apache.spark.sql.catalyst.expressions.UnsafeRow.getUTF8String(UnsafeRow.java:419)
at
org.apache.spark.sql.execution.columnar.STRING$.getField(ColumnType.scala:452)
at
org.apache.spark.sql.execution.columnar.STRING$.getField(ColumnType.scala:424)
at
org.apache.spark.sql.execution.columnar.compression.RunLengthEncoding$Encoder.gatherCompressibilityStats(compressionSchemes.scala:194)
at
org.apache.spark.sql.execution.columnar.compression.CompressibleColumnBuilder$$anonfun$gatherCompressibilityStats$1.apply(CompressibleColumnBuilder.scala:74)
at
org.apache.spark.sql.execution.columnar.compression.CompressibleColumnBuilder$$anonfun$gatherCompressibilityStats$1.apply(CompressibleColumnBuilder.scala:74)
at scala.collection.immutable.List.foreach(List.scala:392) at
org.apache.spark.sql.execution.columnar.compression.CompressibleColumnBuilder$class.gatherCompressibilityStats(CompressibleColumnBuilder.scala:74)

I am processing 100 GB of data with 10 executors of 14G. I startted with
12G executors and I get the same error even with 14G and 3G Overhead memory.
Thanks,
Nimmi

Mime
View raw message