spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Cheng Lian <lian.cs....@gmail.com>
Subject Re: Spark-sql query got exception.Help
Date Wed, 25 Mar 2015 11:05:25 GMT
Could you please provide the full stack trace?

On 3/25/15 6:26 PM, 李铖 wrote:
> It is ok when I do query data from a small hdfs file.
> But if the hdfs file is 152m,I got this exception.
> I try this code 
> .'sc.setSystemProperty("spark.kryoserializer.buffer.mb",'256')'.error 
> still.
>
> ```
> com.esotericsoftware.kryo.KryoException: Buffer overflow. Available: 
> 0, required: 39135
> at com.esotericsoftware.kryo.io.Output.require(Output.java:138)
> at com.esotericsoftware.kryo.io.Output.writeBytes(Output.java:220)
> at com.esotericsoftware.kryo.io.Output.writeBytes(Output.java:206)
> at 
> com.esotericsoftware.kryo.serializers.DefaultArraySerializers$ByteArraySerializer.write(DefaultArraySerializers.java:29)
> at
>
>
> ```


Mime
View raw message