flink-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Nicholas Jiang (Jira)" <j...@apache.org>
Subject [jira] [Commented] (FLINK-14956) MemoryMappedBoundedData Compressed Buffer Slicer
Date Wed, 27 Nov 2019 02:12:00 GMT

    [ https://issues.apache.org/jira/browse/FLINK-14956?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16983093#comment-16983093
] 

Nicholas Jiang commented on FLINK-14956:
----------------------------------------

Hi [~ykt836], I think this is similar to  [FLINK-14845|https://issues.apache.org/jira/browse/FLINK-14845],
I don't know the implementation of this is the same.

> MemoryMappedBoundedData Compressed Buffer Slicer
> ------------------------------------------------
>
>                 Key: FLINK-14956
>                 URL: https://issues.apache.org/jira/browse/FLINK-14956
>             Project: Flink
>          Issue Type: Improvement
>          Components: Runtime / Network
>            Reporter: Nicholas Jiang
>            Priority: Minor
>         Attachments: CPU-IO.png, Compress-Read.png, Compress-Write.png
>
>
> MemoryMappedBoundedData, implementation of BoundedData simply through ByteBuffers backed
by memory, uses CompressedBufferSlicer which is implementation of BoundedData.Reader to slice
next buffer with uncompress. CompressedBufferSlicer reads BoundedData by LZ4SafeDecompressor decompressing
byte buffer.When FileChannelMemoryMappedBoundedData tries to write buffer, this uses LZ4Compressor
to compress buffer to improve I/O performance.
> Compress read process:
> !Compress-Read.png|width=556,height=251!
> Compress write process:
> !Compress-Write.png|width=278,height=261!
> CPU/IO performance comparsion chart:
>   !CPU-IO.png|width=416,height=312!



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Mime
View raw message