ignite-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ilya Kasnacheev <ilya.kasnach...@gmail.com>
Subject Re: Compression prototype
Date Thu, 23 Aug 2018 12:12:50 GMT
Hello!

Currently, the dictionary for decompression is only stored on heap. After
restart there's compressed data in the PDS, but there's no dictionary :)

Regards,

-- 
Ilya Kasnacheev

2018-08-23 14:58 GMT+03:00 Dmitriy Pavlov <dpavlov.spb@gmail.com>:

> Hi Ilya,
>
> Thank you for sharing this here. I believe this contribution will be
> accepted by the Community. Moreover, it shows so remarkable performance
> boost.
>
> I'm pretty sure this patch will be reviewed by Ignite Native Persistence
> experts soon.
>
> What do you mean by can't survive PDS node restart?
>
> Sincerely,
> Dmitriy Pavlov
>
> чт, 23 авг. 2018 г. в 13:09, Ilya Kasnacheev <ilya.kasnacheev@gmail.com>:
>
> > Hello!
> >
> > I want to share with the developer community my compression prototype.
> >
> > Long story short, it compresses BinaryObject's byte[] as they are written
> > to Durable Memory page, operating on a pre-built dictionary. Typical
> > compression ratio is 0.4 (meaning 2.5x compression) using custom
> > LZW+Huffman. Metadata, indexes and primitive values are unaffected
> > entirely.
> >
> > This is akin to DB2's table-level compression[1] but independently
> > invented.
> >
> > On Yardstick tests performance hit is -6% with PDS and up to -25% (in
> > throughput) with In-Memory loads. It also means you can fit ~twice as
> much
> > data into the same IM cluster, or have higher ram/disk ratio with PDS
> > cluster, saving on hardware or decreasing latency.
> >
> > The code is available as PR 4295[2] (set IGNITE_ENABLE_COMPRESSION=true
> to
> > activate). Note that it will not presently survive a PDS node restart.
> > The impact is very small, the patch should be applicable to most 2.x
> > releases.
> >
> > Sure there's a long way before this prototype can have hope of being
> > included, but first I would like to hear input from fellow igniters.
> >
> > See also IEP-20[3].
> >
> > 1.
> >
> > https://www.ibm.com/support/knowledgecenter/en/SSEPGG_10.
> 5.0/com.ibm.db2.luw.admin.dbobj.doc/doc/c0052331.html
> > 2. https://github.com/apache/ignite/pull/4295
> > 3.
> >
> > https://cwiki.apache.org/confluence/display/IGNITE/IEP-
> 20%3A+Data+Compression+in+Ignite
> >
> > Regards,
> >
> > --
> > Ilya Kasnacheev
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message