spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From java8964 <java8...@hotmail.com>
Subject Spark SQL query AVRO file
Date Fri, 07 Aug 2015 18:30:01 GMT
Hi, Spark users:
We currently are using Spark 1.2.2 + Hive 0.12 + Hadoop 2.2.0 on our production cluster, which
has 42 data/task nodes.
There is one dataset stored as Avro files about 3T. Our business has a complex query running
for the dataset, which is stored in nest structure with Array of Struct in Avro and Hive.
We can query it using Hive without any problem, but we like the SparkSQL's performance, so
we in fact run the same query in the Spark SQL, and found out it is in fact much faster than
Hive.
But when we run it, we got the following error randomly from Spark executors, sometime seriously
enough to fail the whole spark job.
Below the stack trace, and I think it is a bug related to Spark due to:
1) The error jumps out inconsistent, as sometimes we won't see it for this job. (We run it
daily)2) Sometime it won't fail our job, as it recover after retry.3) Sometime it will fail
our job, as I listed below.4) Is this due to the multithreading in Spark? The NullPointException
indicates Hive got a Null ObjectInspector of the children of StructObjectInspector, as I read
the Hive source code, but I know there is no null of ObjectInsepector as children of StructObjectInspector.
Google this error didn't give me any hint. Does any one know anything like this?
Project [HiveGenericUdf#org.apache.hadoop.hive.ql.udf.generic.GenericUDFConcatWS(,,CAST(account_id#23L,
StringType),CAST(gross_contact_count_a#4L, StringType),CASE WHEN IS NULL tag_cnt#21 THEN 0
ELSE CAST(tag_cnt#21, StringType),CAST(list_cnt_a#5L, StringType),CAST(active_contact_count_a#16L,
StringType),CAST(other_api_contact_count_a#6L, StringType),CAST(fb_api_contact_count_a#7L,
StringType),CAST(evm_contact_count_a#8L, StringType),CAST(loyalty_contact_count_a#9L, StringType),CAST(mobile_jmml_contact_count_a#10L,
StringType),CAST(savelocal_contact_count_a#11L, StringType),CAST(siteowner_contact_count_a#12L,
StringType),CAST(socialcamp_service_contact_count_a#13L, S...org.apache.spark.SparkException:
Job aborted due to stage failure: Task 58 in stage 1.0 failed 4 times, most recent failure:
Lost task 58.3 in stage 1.0 (TID 257, 10.20.95.146): java.lang.NullPointerException      
 at org.apache.hadoop.hive.serde2.avro.AvroObjectInspectorGenerator.supportedCategories(AvroObjectInspectorGenerator.java:139)
       at org.apache.hadoop.hive.serde2.avro.AvroObjectInspectorGenerator.createObjectInspectorWorker(AvroObjectInspectorGenerator.java:89)
       at org.apache.hadoop.hive.serde2.avro.AvroObjectInspectorGenerator.createObjectInspectorWorker(AvroObjectInspectorGenerator.java:101)
       at org.apache.hadoop.hive.serde2.avro.AvroObjectInspectorGenerator.createObjectInspectorWorker(AvroObjectInspectorGenerator.java:117)
       at org.apache.hadoop.hive.serde2.avro.AvroObjectInspectorGenerator.createObjectInspector(AvroObjectInspectorGenerator.java:81)
       at org.apache.hadoop.hive.serde2.avro.AvroObjectInspectorGenerator.<init>(AvroObjectInspectorGenerator.java:55)
       at org.apache.hadoop.hive.serde2.avro.AvroSerDe.initialize(AvroSerDe.java:69)     
  at org.apache.spark.sql.hive.HadoopTableReader$$anonfun$2.apply(TableReader.scala:112) 
      at org.apache.spark.sql.hive.HadoopTableReader$$anonfun$2.apply(TableReader.scala:109)
       at org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:618)        at org.apache.spark.rdd.RDD$$anonfun$14.apply(RDD.scala:618)
       at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)       
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:280)        at org.apache.spark.rdd.RDD.iterator(RDD.scala:247)
       at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)       
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:280)        at org.apache.spark.rdd.RDD.iterator(RDD.scala:247)
       at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)       
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:280)        at org.apache.spark.rdd.RDD.iterator(RDD.scala:247)
       at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)       
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:280)        at org.apache.spark.rdd.RDD.iterator(RDD.scala:247)
       at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)       
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:280)        at org.apache.spark.rdd.RDD.iterator(RDD.scala:247)
       at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:68)     
  at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:41)        at
org.apache.spark.scheduler.Task.run(Task.scala:56)        at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:198)
       at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
       at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
       at java.lang.Thread.run(Thread.java:745)h         		 	   		  
Mime
View raw message