ignite-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sergey Kozlov <skoz...@gridgain.com>
Subject Re: Compression prototype
Date Thu, 23 Aug 2018 12:13:40 GMT
Hi Ilya

Is there a plan to introduce it as an option of Ignite configuration? In
that instead the boolean type I suggest to use the enum and reserve the
ability to extend compressions algorithms in future

On Thu, Aug 23, 2018 at 1:09 PM, Ilya Kasnacheev <ilya.kasnacheev@gmail.com>
wrote:

> Hello!
>
> I want to share with the developer community my compression prototype.
>
> Long story short, it compresses BinaryObject's byte[] as they are written
> to Durable Memory page, operating on a pre-built dictionary. Typical
> compression ratio is 0.4 (meaning 2.5x compression) using custom
> LZW+Huffman. Metadata, indexes and primitive values are unaffected
> entirely.
>
> This is akin to DB2's table-level compression[1] but independently
> invented.
>
> On Yardstick tests performance hit is -6% with PDS and up to -25% (in
> throughput) with In-Memory loads. It also means you can fit ~twice as much
> data into the same IM cluster, or have higher ram/disk ratio with PDS
> cluster, saving on hardware or decreasing latency.
>
> The code is available as PR 4295[2] (set IGNITE_ENABLE_COMPRESSION=true to
> activate). Note that it will not presently survive a PDS node restart.
> The impact is very small, the patch should be applicable to most 2.x
> releases.
>
> Sure there's a long way before this prototype can have hope of being
> included, but first I would like to hear input from fellow igniters.
>
> See also IEP-20[3].
>
> 1.
> https://www.ibm.com/support/knowledgecenter/en/SSEPGG_10.
> 5.0/com.ibm.db2.luw.admin.dbobj.doc/doc/c0052331.html
> 2. https://github.com/apache/ignite/pull/4295
> 3.
> https://cwiki.apache.org/confluence/display/IGNITE/IEP-
> 20%3A+Data+Compression+in+Ignite
>
> Regards,
>
> --
> Ilya Kasnacheev
>



-- 
Sergey Kozlov
GridGain Systems
www.gridgain.com

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message