spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jörn Franke <jornfra...@gmail.com>
Subject Re: AVRO File size when caching in-memory
Date Mon, 14 Nov 2016 09:36:16 GMT
spark version? Are you using tungsten?

> On 14 Nov 2016, at 10:05, Prithish <prithish@gmail.com> wrote:
> 
> Can someone please explain why this happens?
> 
> When I read a 600kb AVRO file and cache this in memory (using cacheTable), it shows up
as 11mb (storage tab in Spark UI). I have tried this with different file sizes, and the size
in-memory is always proportionate. I thought Spark compresses when using cacheTable. 

---------------------------------------------------------------------
To unsubscribe e-mail: user-unsubscribe@spark.apache.org


Mime
View raw message