hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Steve Loughran (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-16228) Throwing OOM exception for ListStatus (v2) when using S3A
Date Thu, 04 Apr 2019 20:40:00 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-16228?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16810267#comment-16810267

Steve Loughran commented on HADOOP-16228:

This looks like a bug in the AWS SDK, something in its XML parsing of the LIST response

* Is this a genuine AWS S3 endpoint, or someone else's s3-compatible one?
* is there something "odd" about the files in your directory tree (i.e. unusual characters
in them)
* if you install the aws sdk, does an {{aws s3 ls}} command work?
* Does this still happen on the Hadoop 3.2 release?
* Is it only one specific bucket/path?

Given it's an AWS-side problem, we can't fix it, but since it's getting time to update the
AWS SDK on trunk, maybe you could help us see if this problem goes away from you then.

otherwise, without that update, I'm afraid you'll have to turn v2 listing off, at least, unless
we can find some other root cause.

One trick would be actually grab the HTTP level logs, see: https://hadoop.apache.org/docs/current3/hadoop-aws/tools/hadoop-aws/troubleshooting_s3a.html#Enabling_low-level_logging

I don't want those logs as they'll inevitably contain sensitive filenames or something, but
if you can see anything obvious in the response (like you don't get XML back), that would
help track this down

> Throwing OOM exception for ListStatus (v2) when using S3A 
> ----------------------------------------------------------
>                 Key: HADOOP-16228
>                 URL: https://issues.apache.org/jira/browse/HADOOP-16228
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs/s3
>    Affects Versions: 3.1.0
>            Reporter: Ranith Sardar
>            Priority: Major
> {code:java}
>  @InterfaceStability.Unstable
>   public static final String LIST_VERSION = "fs.s3a.list.version";
>   @InterfaceStability.Unstable
>   public static final int DEFAULT_LIST_VERSION = 2;
> {code}
> If the files in the bucket are more than 1k, it will throw OOM error for V2 version.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org

View raw message