hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jameson Li <hovlj...@gmail.com>
Subject hbase host dns ip and route for multi network interface card
Date Tue, 05 Jul 2011 10:13:17 GMT
Hi,

when I start my hbase cluster, there are some error logs in the master-log:
<the ip and hostname node3 192.168.1.15 192.168.1.13 are the same machine
that have two NIC>
2011-07-05 17:13:13,820 INFO org.apache.zookeeper.ClientCnxn:
zookeeper.disableAutoWatchReset is false
2011-07-05 17:13:13,840 INFO org.apache.zookeeper.ClientCnxn: Attempting
connection to server node3/192.168.1.15:2181
....
2011-07-05 17:13:13,975 DEBUG org.apache.hadoop.hbase.master.HMaster:
Checking cluster state...
2011-07-05 17:13:13,979 DEBUG
org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper: Read ZNode
/hbase/root-region-server got 192.168.1.13:60020
....
2011-07-05 17:13:19,732 DEBUG
org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper: Updated ZNode
/hbase/rs/1309857199677 with data 192.168.1.15:60020
....
2011-07-05 17:22:01,041 INFO org.apache.hadoop.ipc.HbaseRPC: Server at /
192.168.1.13:60020 could not be reached after 1 tries, giving up.
2011-07-05 17:22:01,042 WARN org.apache.hadoop.hbase.master.BaseScanner:
Scan one META region: {server: 192.168.1.13:60020, regionname: .META.,,1,
startKey: <>}org.apache.hadoop.hbase.client.RetriesExhaustedException:
Failed setting up proxy to /192.168.1.13:60020 after attempts=1
        at
org.apache.hadoop.hbase.ipc.HBaseRPC.waitForProxy(HBaseRPC.java:429)
        at
org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getHRegionConnection(HConnectionManager.java:918)
        at
org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getHRegionConnection(HConnectionManager.java:934)
        at
org.apache.hadoop.hbase.master.BaseScanner.scanRegion(BaseScanner.java:173)
        at
org.apache.hadoop.hbase.master.MetaScanner.scanOneMetaRegion(MetaScanner.java:73)
        at
org.apache.hadoop.hbase.master.MetaScanner.maintenanceScan(MetaScanner.java:129)
        at
org.apache.hadoop.hbase.master.BaseScanner.chore(BaseScanner.java:153)
        at org.apache.hadoop.hbase.Chore.run(Chore.java:68)

Sometimes when the .META. region is not assigned to the server node3, which
has two NIC:eth0:192.168.1.13 and eth1:192.168.1.15 and resolve the dns/host
as:192.168.1.15 node3, I means, when the region .META. is assigned to the
others server that has only one NIC, the hbase will work well.

here is some of my hbase cluster infos:
Hbase version:0.20.6
Hadoop version:0.20-append+4
Zookeeper version:3.3.0

the hbase-site.xml:
<configuration>
<property>
<name>hbase.rootdir</name>
<value>hdfs://node3:54310/hbase</value>
</property>

<property>
<name>hbase.master</name>
<value>hadoop5:60000</value>
</property>

<property>
<name>hbase.zookeeper.quorum</name>
<value>node3,hadoop5,hadoopoffice85,hadoopoffice88,hdofficelj001</value>
</property>

<property>
    <name>hbase.cluster.distributed</name>
    <value>true</value>
  </property>

  <!--property>
    <name>hbase.master.dns.interface</name>
    <value>eth1</value>
    <description>The name of the Network Interface from which a master
      should report its IP address.
    </description>
  </property>

<property>
    <name>hbase.regionserver.dns.interface</name>
    <value>eth1</value>
    <description>The name of the Network Interface from which a region
server
      should report its IP address.
    </description>
  </property>

<property>
    <name>hbase.zookeeper.dns.interface</name>
    <value>eth1</value>
    <description>The name of the Network Interface from which a ZooKeeper
server
      should report its IP address.
    </description>
  </property-->

<!--property>
      <name>hbase.zookeeper.property.clientPort</name>
      <value>2222</value>
      <description>Property from ZooKeeper's config zoo.cfg.
      The port at which the clients will connect.
      </description>
    </property>
<property>
      <name>hbase.zookeeper.property.dataDir</name>
      <value>/opt/zookeeper/data</value>
      <description>Property from ZooKeeper's config zoo.cfg.
      The directory where the snapshot is stored.
      </description>
    </property-->
</configuration>

cat /opt/hbase/conf/regionservers
hadoop5
node3
hadoopoffice85
hadoopoffice88
hdofficelj001
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------
And the below is the node3's info:
192.168.1.13's ifconfig info:
[root@node3 ~]# ifconfig
eth0      Link encap:Ethernet  HWaddr 00:0C:29:23:2E:D3
          inet addr:192.168.1.13  Bcast:192.168.1.255  Mask:255.255.255.0
          inet6 addr: fe80::20c:29ff:fe23:2ed3/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:1424620 errors:0 dropped:0 overruns:0 frame:0
          TX packets:17897973 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:150231810 (143.2 MiB)  TX bytes:2834085782 (2.6 GiB)
          Base address:0x2000 Memory:d8920000-d8940000

eth1      Link encap:Ethernet  HWaddr 00:0C:29:23:2E:DD
          inet addr:192.168.1.15  Bcast:192.168.1.255  Mask:255.255.255.0
          inet6 addr: fe80::20c:29ff:fe23:2edd/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:1172226 errors:0 dropped:0 overruns:0 frame:0
          TX packets:1445 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:168873362 (161.0 MiB)  TX bytes:293447 (286.5 KiB)
          Base address:0x2040 Memory:d8940000-d8960000

lo        Link encap:Local Loopback
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:16436  Metric:1
          RX packets:370550 errors:0 dropped:0 overruns:0 frame:0
          TX packets:370550 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:64864387 (61.8 MiB)  TX bytes:64864387 (61.8 MiB)

the hosts info:
[root@node3 ~]# cat /etc/hosts
# Do not remove the following line, or various programs
# that require network functionality will fail.
127.0.0.1 localhost.localdomain localhost
::1 localhost6.localdomain6 localhost6
192.168.1.27 hadoop5
192.168.1.15 node3
192.168.1.85 hadoopoffice85
192.168.1.88 hadoopoffice88
192.168.3.227 hdofficelj001

[root@node3 ~]# netstat -nap | grep 600
tcp        0      0 ::ffff:192.168.1.15:60020   :::*
 LISTEN      19064/java
tcp        0      0 :::60030                    :::*
 LISTEN      19064/java
tcp        0      0 ::ffff:192.168.1.13:44350   ::ffff:192.168.1.27:60000
ESTABLISHED 19064/java

[root@node3 ~]# route
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use
Iface
239.2.11.71     *               255.255.255.255 UH    0      0        0 eth1
192.168.1.0     *               255.255.255.0   U     0      0        0 eth0
192.168.1.0     *               255.255.255.0   U     0      0        0 eth1
169.254.0.0     *               255.255.0.0     U     0      0        0 eth1
default         192.168.1.254   0.0.0.0         UG    0      0        0 eth0
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------
I have added the dns.interface configuration as eth1, but it still has the
same error.
<property>
    <name>hbase.master.dns.interface</name>
    <value>eth1</value>
    <description>The name of the Network Interface from which a master
      should report its IP address.
    </description>
  </property>

<property>
    <name>hbase.regionserver.dns.interface</name>
    <value>eth1</value>
    <description>The name of the Network Interface from which a region
server
      should report its IP address.
    </description>
  </property>

<property>
    <name>hbase.zookeeper.dns.interface</name>
    <value>eth1</value>
    <description>The name of the Network Interface from which a ZooKeeper
server
      should report its IP address.
    </description>
  </property>


---------------------------------------------------------------------------------------------------------------------------------------------------------------------------
But After I change the default route, but it still has the same error.
[root@node3 ~]# route del default
[root@node3 ~]# route add -net default gw 192.168.1.254 eth1
[root@node3 ~]# route
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use
Iface
239.2.11.71     *               255.255.255.255 UH    0      0        0 eth1
192.168.1.0     *               255.255.255.0   U     0      0        0 eth0
192.168.1.0     *               255.255.255.0   U     0      0        0 eth1
169.254.0.0     *               255.255.0.0     U     0      0        0 eth1
default         192.168.1.254   0.0.0.0         UG    0      0        0 eth1
[root@node3 ~]# netstat -nap | grep 600
tcp        0      0 ::ffff:192.168.1.15:60020   :::*
 LISTEN      23282/java
tcp        0      0 :::60030                    :::*
 LISTEN      23282/java
tcp        0      0 ::ffff:192.168.1.13:45783   ::ffff:192.168.1.27:60000
ESTABLISHED 23282/java

Help.

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message