flink-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Nicholas Jiang (Jira)" <j...@apache.org>
Subject [jira] [Comment Edited] (FLINK-14956) MemoryMappedBoundedData Compressed Buffer Slicer
Date Wed, 27 Nov 2019 02:27:00 GMT

    [ https://issues.apache.org/jira/browse/FLINK-14956?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16983093#comment-16983093

Nicholas Jiang edited comment on FLINK-14956 at 11/27/19 2:26 AM:

Hi [~ykt836], I think the idea is similar to  [FLINK-14845|https://issues.apache.org/jira/browse/FLINK-14845],
the implementation of improvement is the same. Therefore, [~kevin.cyj], I follow your implementation
of compress.

was (Author: nicholasjiang):
Hi [~ykt836], I think this is similar to  [FLINK-14845|https://issues.apache.org/jira/browse/FLINK-14845],
I don't know the implementation of this is the same.

> MemoryMappedBoundedData Compressed Buffer Slicer
> ------------------------------------------------
>                 Key: FLINK-14956
>                 URL: https://issues.apache.org/jira/browse/FLINK-14956
>             Project: Flink
>          Issue Type: Improvement
>          Components: Runtime / Network
>            Reporter: Nicholas Jiang
>            Priority: Minor
>         Attachments: CPU-IO.png, Compress-Read.png, Compress-Write.png
> MemoryMappedBoundedData, implementation of BoundedData simply through ByteBuffers backed
by memory, uses CompressedBufferSlicer which is implementation of BoundedData.Reader to slice
next buffer with uncompress. CompressedBufferSlicer reads BoundedData by LZ4SafeDecompressor decompressing
byte buffer.When FileChannelMemoryMappedBoundedData tries to write buffer, this uses LZ4Compressor
to compress buffer to improve I/O performance.
> Compress read process:
> !Compress-Read.png|width=556,height=251!
> Compress write process:
> !Compress-Write.png|width=278,height=261!
> CPU/IO performance comparsion chart:
>   !CPU-IO.png|width=416,height=312!

This message was sent by Atlassian Jira

View raw message