spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From ÐΞ€ρ@Ҝ (๏̯͡๏) <deepuj...@gmail.com>
Subject Spark program throws NIO Buffer over flow error (TDigest - Ted Dunning lib)
Date Thu, 09 Jul 2015 01:50:03 GMT
Any suggestions ?
Code:

    val dimQuantiles = genericRecordsAndKeys

      .map {

        case (keyToOutput, rec) =>

          var digest: TDigest = TDigest.createAvlTreeDigest(10000)

          val fpPaidGMB = rec.get("fpPaidGMB").asInstanceOf[Double]

          digest.add(fpPaidGMB)

          var bbuf: ByteBuffer = ByteBuffer.allocate(digest.byteSize());

          digest.asBytes(bbuf);

          (keyToOutput.toString, bbuf.array())

      }


Trace:
aborted due to stage failure: Task 4487 in stage 1.0 failed 4 times, most
recent failure: Lost task 4487.3 in stage 1.0 (TID 281,
phxaishdc9dn0111.phx.ebay.com): java.nio.BufferOverflowException
at java.nio.Buffer.nextPutIndex(Buffer.java:519)
at java.nio.HeapByteBuffer.putDouble(HeapByteBuffer.java:519)
at com.tdunning.math.stats.AVLTreeDigest.asBytes(AVLTreeDigest.java:336)
at
com.ebay.ep.poc.spark.reporting.process.service.OutlierService$$anonfun$4.apply(OutlierService.scala:96)
at
com.ebay.ep.poc.spark.reporting.process.service.OutlierService$$anonfun$4.apply(OutlierService.scala:89)
at
org.apache.spark.util.collection.ExternalSorter$$anonfun$5.apply(ExternalSorter.scala:197)
at
org.apache.spark.util.collection.ExternalSorter$$anonfun$5.apply(ExternalSorter.scala:196)
at
org.apache.spark.util.collection.AppendOnlyMap.changeValue(AppendOnlyMap.scala:138)
at
org.apache.spark.util.collection.SizeTrackingAppendOnlyMap.changeValue(SizeTrackingAppendOnlyMap.scala:32)
at
org.apache.spark.util.collection.ExternalSorter.insertAll(ExternalSorter.scala:202)
at
org.apache.spark.shuffle.sort.SortShuffleWriter.write(SortShuffleWriter.scala:56)
at
org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:70)
at
org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:41)


-- 
Deepak

Mime
View raw message