spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From nitinkak001 <nitinkak...@gmail.com>
Subject com.esotericsoftware.kryo.KryoException: Buffer overflow.
Date Tue, 21 Oct 2014 21:30:27 GMT
I am running a simple rdd filter command. What does it mean? 
Here is the full stack trace(and code below it): 

com.esotericsoftware.kryo.KryoException: Buffer overflow. Available: 0,
required: 133
        at com.esotericsoftware.kryo.io.Output.require(Output.java:138)
        at
com.esotericsoftware.kryo.io.Output.writeString_slow(Output.java:420)
        at com.esotericsoftware.kryo.io.Output.writeString(Output.java:326)
        at
com.esotericsoftware.kryo.serializers.DefaultArraySerializers$StringArraySerializer.write(DefaultArraySerializers.java:274)
        at
com.esotericsoftware.kryo.serializers.DefaultArraySerializers$StringArraySerializer.write(DefaultArraySerializers.java:262)
        at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:568)
        at
org.apache.spark.serializer.KryoSerializerInstance.serialize(KryoSerializer.scala:138)
        at
org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:197)
        at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:744)

*Here is the code of the main function:*

/String comparisonFieldIndexes = "16,18";
String segmentFieldIndexes = "14,15";
String comparisonFieldWeights = "50, 50";
String delimiter = ""+'\001';
		
PartitionDataOnColumn parOnCol = new PartitionDataOnColumn(70,
comparisonFieldIndexes, comparisonFieldWeights, segmentFieldIndexes,
delimiter);
		
JavaRDD<String> filtered_rdd = origRDD.filter(parOnCol.new
FilterEmptyFields(parOnCol.fieldIndexes, parOnCol.DELIMITER) );
	    
parOnCol.printRDD(filtered_rdd);/


*Here is the FilterEmptyFields class:*

        /public class FilterEmptyFields implements Function<String, Boolean>
{
		
		final int[] nonEmptyFields;
		final String DELIMITER;
		
		public FilterEmptyFields(int[] nonEmptyFields, String delimiter){
			this.nonEmptyFields = nonEmptyFields;
			this.DELIMITER = delimiter;
		}
		
		@Override
		public Boolean call(String s){
			
			String[] fields = s.split(DELIMITER);
			
			for(int i=0; i<nonEmptyFields.length; i++){
				if(fields[nonEmptyFields[i]] == null  ||
fields[nonEmptyFields[i]].isEmpty()){
					return false;
				}
			}
			
			return true;
		}
		
	}&lt;/i>

Any suggestions guys?



--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/com-esotericsoftware-kryo-KryoException-Buffer-overflow-tp16947.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org


Mime
View raw message