spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Wangchangchun (A)" <>
Subject [ compress in-memory column storage used in sparksql cache table ]
Date Wed, 02 Sep 2015 04:23:41 GMT
Hi,  I have an idea, can someone give me some advice?

I want to compress data in in-memory column storage which is used by cache table in spark.
This will make cache table use less memory.

I will set an conf to this function, so if anyone want to use this function, he can set this
conf to true.

Compress algorithom I want to use Dictionary Encoding.

Do you think this method worth a try ?

View raw message