hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Junping Du (JIRA)" <j...@apache.org>
Subject [jira] [Created] (HADOOP-8468) Umbrella of enhancements to support different failure and locality topologies
Date Mon, 04 Jun 2012 04:09:22 GMT
Junping Du created HADOOP-8468:

             Summary: Umbrella of enhancements to support different failure and locality topologies
                 Key: HADOOP-8468
                 URL: https://issues.apache.org/jira/browse/HADOOP-8468
             Project: Hadoop Common
          Issue Type: Bug
          Components: ha, io
    Affects Versions: 2.0.0-alpha, 1.0.0
            Reporter: Junping Du
            Assignee: Junping Du
            Priority: Critical

The current hadoop network topology (described in some previous issues like: Hadoop-692) works
well in classic three-tiers network when it comes out. However, it does not take into account
other failure models or changes in the infrastructure that can affect network bandwidth efficiency
like: virtualization. 
Virtualized platform has following genes that shouldn't been ignored by hadoop topology in
scheduling tasks, placing replica, do balancing or fetching block for reading: 
1. VMs on the same physical host are affected by the same hardware failure. In order to match
the reliability of a physical deployment, replication of data across two virtual machines
on the same host should be avoided.
2. The network between VMs on the same physical host has higher throughput and lower latency
and does not consume any physical switch bandwidth.
Thus, we propose to make hadoop network topology extend-able and introduce a new level in
the hierarchical topology, a node group level, which maps well onto an infrastructure that
is based on a virtualized environment.

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira


View raw message