spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Koert Kuipers <ko...@tresata.com>
Subject Re: com.esotericsoftware.kryo.KryoException: Buffer overflow.
Date Wed, 22 Oct 2014 02:13:47 GMT
you ran out of kryo buffer. are you using spark 1.1 (which supports buffer
resizing) or spark 1.0 (which has a fixed size buffer)?
On Oct 21, 2014 5:30 PM, "nitinkak001" <nitinkak001@gmail.com> wrote:

> I am running a simple rdd filter command. What does it mean?
> Here is the full stack trace(and code below it):
>
> com.esotericsoftware.kryo.KryoException: Buffer overflow. Available: 0,
> required: 133
>         at com.esotericsoftware.kryo.io.Output.require(Output.java:138)
>         at
> com.esotericsoftware.kryo.io.Output.writeString_slow(Output.java:420)
>         at com.esotericsoftware.kryo.io.Output.writeString(Output.java:326)
>         at
>
> com.esotericsoftware.kryo.serializers.DefaultArraySerializers$StringArraySerializer.write(DefaultArraySerializers.java:274)
>         at
>
> com.esotericsoftware.kryo.serializers.DefaultArraySerializers$StringArraySerializer.write(DefaultArraySerializers.java:262)
>         at
> com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:568)
>         at
>
> org.apache.spark.serializer.KryoSerializerInstance.serialize(KryoSerializer.scala:138)
>         at
> org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:197)
>         at
>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>         at
>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>         at java.lang.Thread.run(Thread.java:744)
>
> *Here is the code of the main function:*
>
> /String comparisonFieldIndexes = "16,18";
> String segmentFieldIndexes = "14,15";
> String comparisonFieldWeights = "50, 50";
> String delimiter = ""+'\001';
>
> PartitionDataOnColumn parOnCol = new PartitionDataOnColumn(70,
> comparisonFieldIndexes, comparisonFieldWeights, segmentFieldIndexes,
> delimiter);
>
> JavaRDD<String> filtered_rdd = origRDD.filter(parOnCol.new
> FilterEmptyFields(parOnCol.fieldIndexes, parOnCol.DELIMITER) );
>
> parOnCol.printRDD(filtered_rdd);/
>
>
> *Here is the FilterEmptyFields class:*
>
>         /public class FilterEmptyFields implements Function<String,
> Boolean>
> {
>
>                 final int[] nonEmptyFields;
>                 final String DELIMITER;
>
>                 public FilterEmptyFields(int[] nonEmptyFields, String
> delimiter){
>                         this.nonEmptyFields = nonEmptyFields;
>                         this.DELIMITER = delimiter;
>                 }
>
>                 @Override
>                 public Boolean call(String s){
>
>                         String[] fields = s.split(DELIMITER);
>
>                         for(int i=0; i<nonEmptyFields.length; i++){
>                                 if(fields[nonEmptyFields[i]] == null  ||
> fields[nonEmptyFields[i]].isEmpty()){
>                                         return false;
>                                 }
>                         }
>
>                         return true;
>                 }
>
>         }&lt;/i>
>
> Any suggestions guys?
>
>
>
> --
> View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/com-esotericsoftware-kryo-KryoException-Buffer-overflow-tp16947.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
>
>

Mime
View raw message