hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sahil Takiar (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-14603) S3A input stream to support ByteBufferReadable
Date Mon, 18 Feb 2019 16:16:00 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-14603?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16771190#comment-16771190
] 

Sahil Takiar commented on HADOOP-14603:
---------------------------------------

Looked into this a bit while working on HDFS-3246, as far as I can tell, the AWS SDK does
not support {{ByteBuffer}} only {{byte[]}}, which will makes this optimization less useful
since passing in a {{DirectByteBuffer}} would require copying the data into a {{byte []}}
before it can be passed to the AWS SDK. It looks like the AWS SDK is just reading data from
a {{SocketInputStream}} rather than a {{SocketChannel}}. Whereas HDFS reads from either a
{{ReadableByteChannel}} for remote blocks, or a {{FileChannel}} for lock blocks.

I'm not sure if the AWS SDK plans to migrate to {{ByteBuffer}} anytime soon. [This|https://aws.amazon.com/blogs/developer/aws-sdk-for-java-2-x-released/]
blog post about the 2.x AWS SDK suggests the 2.x SDK has {{ByteBuffer}} support, but I haven't
looked into it in detail / not sure if we plan to upgrade to the new SDK anytime soon.

> S3A input stream to support ByteBufferReadable
> ----------------------------------------------
>
>                 Key: HADOOP-14603
>                 URL: https://issues.apache.org/jira/browse/HADOOP-14603
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: fs/s3, test
>    Affects Versions: 2.8.1
>            Reporter: Steve Loughran
>            Priority: Minor
>
> S3AInputStream could support {{ByteBufferReadable, HasEnhancedByteBufferAccess}} and
the operations to read into byte buffers.
> This is only if we can see a clear performance benefit from doing this or the API is
being more broadly used



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


Mime
View raw message