hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ted Yu <yuzhih...@gmail.com>
Subject Re: RegionServer continues lease expired
Date Sat, 07 Mar 2015 14:33:47 GMT
bq. once it receives the first region

Looks like it was the former case Nick mentioned.

bq. this slave node is identified with its FQDN

Michele:
Have you tried to make the setup of this slave node identical to the others
(using hostname no domain) ?

Cheers

On Fri, Mar 6, 2015 at 10:23 PM, Nick Dimiduk <ndimiduk@gmail.com> wrote:

> You get the lease expired from any region hosted on this machine, or only
> regions from a specific table? If the former, I'd say kill the machine and
> request a new one. On AES, life is too short to diagnose crap VM's. If you
> do that and the problem moves to another machine, it must the the region ;)
>
> On Friday, March 6, 2015, Giusto Michele <m.giusto@reply.it> wrote:
>
> >  Hi all,
> >
> > I am working on a Cloudera CDH5.3.1 cluster (so hbase-0.98.6+cdh5.3.1+74)
> > and the cluster is running in AWS. The cluster is composed of 6
> machines, 2
> > masters and 4 slaves. I am having problem on a single slave node (the
> last
> > one added, temporarily) where the local RegionServer, once it receives
> the
> > first region (even without rows inside), continuously logs messages like:
> >
> > 2015-03-06 11:48:29,088 INFO
> > org.apache.hadoop.hbase.regionserver.HRegionServer: Scanner 1688 lease
> > expired on region
> > test_table2,,1425636551770.739b80dfcac4fbc59ea6a6633b9e86a8.
> >
> > 2015-03-06 11:48:32,458 INFO
> > org.apache.hadoop.hbase.regionserver.HRegionServer: Scanner 1689 lease
> > expired on region
> > test_table2,,1425636551770.739b80dfcac4fbc59ea6a6633b9e86a8.
> >
> > 2015-03-06 11:48:36,089 INFO
> > org.apache.hadoop.hbase.regionserver.HRegionServer: Scanner 1690 lease
> > expired on region
> > test_table2,,1425636551770.739b80dfcac4fbc59ea6a6633b9e86a8.
> >
> > 2015-03-06 11:48:39,460 INFO
> > org.apache.hadoop.hbase.regionserver.HRegionServer: Scanner 1691 lease
> > expired on region
> > test_table2,,1425636551770.739b80dfcac4fbc59ea6a6633b9e86a8.
> >
> > 2015-03-06 11:48:43,090 INFO
> > org.apache.hadoop.hbase.regionserver.HRegionServer: Scanner 1692 lease
> > expired on region
> > test_table2,,1425636551770.739b80dfcac4fbc59ea6a6633b9e86a8.
> >
> > 2015-03-06 11:48:46,459 INFO
> > org.apache.hadoop.hbase.regionserver.HRegionServer: Scanner 1693 lease
> > expired on region
> > test_table2,,1425636551770.739b80dfcac4fbc59ea6a6633b9e86a8.
> >
> > 2015-03-06 11:48:50,089 INFO
> > org.apache.hadoop.hbase.regionserver.HRegionServer: Scanner 1694 lease
> > expired on region
> > test_table2,,1425636551770.739b80dfcac4fbc59ea6a6633b9e86a8.
> >
> >
> >
> > All the other services are working as expected apart from this problem. I
> > am not able to understand why this is happening, the only difference I
> can
> > see from logs/web UI is that this slave node is identified with its FQDN,
> > while the others only with the hostname (no domain).
> >
> >
> >
> > I have checked on the other nodes and they are not reporting similar
> > messages. Setting log level to TRACE didn’t show nothing else useful, at
> > least for me.
> >
> >
> >
> > Finally, the original problem that lead me to find these messages is that
> > when I map an HBase table as an external table on Hive/Impala, with one
> > region of this table on the problematic RegionServer, the query remain
> > pending forever. If the table is not using that RegionServer, queries
> work
> > normally. It is important to note that the messages starts immediately
> > after I create the HBase table, so I believe it is not Impala that
> creates
> > problem.
> >
> > What does the messages imply? Can the reason of my Impala problem be
> > related to these messages?
> >
> >
> >
> >
> >
> > Thanks in advance,
> >
> > Michele
> >
> >
> >  Michele Giusto
> >
> > Target Reply
> > Via Castellanza, 11
> > 20151 - Milano - ITALY
> > phone: +39 02 535761
> > m.giusto@reply.it <javascript:_e(%7B%7D,'cvml','m.giusto@reply.it');>
> > www.reply.it
> >
> > [image: Target Reply]
> >
> >
> > ------------------------------
> >
> > --
> > The information transmitted is intended for the person or entity to which
> > it is addressed and may contain confidential and/or privileged material.
> > Any review, retransmission, dissemination or other use of, or taking of
> any
> > action in reliance upon, this information by persons or entities other
> than
> > the intended recipient is prohibited. If you received this in error,
> please
> > contact the sender and delete the material from any computer.
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message