hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Aaron T. Myers (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-9150) Unnecessary DNS resolution attempts for logical URIs
Date Mon, 17 Dec 2012 22:58:32 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-9150?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13534396#comment-13534396
] 

Aaron T. Myers commented on HADOOP-9150:
----------------------------------------

bq. Or, since I think this also affects viewfs and other filesystems which don't strictly
use hostnames (eg s3n://bucketname/), we can add an overridable isLogicalUri() in the FileSystem
interface itself, and then have the affected filesystems return true as necessary.

This one makes sense to me, given that this affects more than just HDFS proper.
                
> Unnecessary DNS resolution attempts for logical URIs
> ----------------------------------------------------
>
>                 Key: HADOOP-9150
>                 URL: https://issues.apache.org/jira/browse/HADOOP-9150
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs/s3, ha, viewfs
>    Affects Versions: 3.0.0, 2.0.2-alpha
>            Reporter: Todd Lipcon
>            Priority: Critical
>         Attachments: log.txt, tracing-resolver.tgz
>
>
> In the FileSystem code, we accidentally try to DNS-resolve the logical name before it
is converted to an actual domain name. In some DNS setups, this can cause a big slowdown -
eg in one misconfigured cluster we saw a 2-3x drop in terasort throughput, since every task
wasted a lot of time waiting for slow "not found" responses from DNS.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message