flink-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Kurt Young (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (FLINK-11775) Introduce MemorySegmentWritable to let DataOutputView direct copy to internal bytes
Date Thu, 11 Apr 2019 08:39:01 GMT

    [ https://issues.apache.org/jira/browse/FLINK-11775?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16815206#comment-16815206
] 

Kurt Young commented on FLINK-11775:
------------------------------------

I also think letting AbstractPagedOutputView implements MemorySegmentWritable (not exposing
to DataOutputView) is more appropriate. From my understanding, the concept of *page* in AbstractPagedOutputView
is similar with MemorySegment we have, or even it is representing MemorySegment, but just
with a different name. Conceptually, it will be more align we let AbstractPagedOutputView
know MemorySegment and have an optimal way to write it.

> Introduce MemorySegmentWritable to let DataOutputView direct copy to internal bytes
> -----------------------------------------------------------------------------------
>
>                 Key: FLINK-11775
>                 URL: https://issues.apache.org/jira/browse/FLINK-11775
>             Project: Flink
>          Issue Type: New Feature
>          Components: Runtime / Operators
>            Reporter: Jingsong Lee
>            Assignee: Jingsong Lee
>            Priority: Major
>
> Blink new binary format is based on MemorySegment.
> Introduce MemorySegmentWritable to let DataOutputView direct copy to internal bytes
> {code:java}
> /**
>  * Provides the interface for write(Segment).
>  */
> public interface MemorySegmentWritable {
>  /**
>  * Writes {@code len} bytes from memory segment {@code segment} starting at offset {@code
off}, in order,
>  * to the output.
>  *
>  * @param segment memory segment to copy the bytes from.
>  * @param off the start offset in the memory segment.
>  * @param len The number of bytes to copy.
>  * @throws IOException if an I/O error occurs.
>  */
>  void write(MemorySegment segment, int off, int len) throws IOException;
> }{code}
>  
> If we want to write a Memory Segment to DataOutputView, we need to copy bytes to byte[]
and then write it in, which is less effective.
> If we let AbstractPagedOutputView have a write(MemorySegment) interface, we can copy
it directly.
> We need to ensure this in network serialization, batch operator calculation serialization,
Streaming State serialization to avoid new byte[] and copy.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message