hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From schubert zhang <zson...@gmail.com>
Subject Re: HRegionServer: Scanner, lease expired issue.
Date Wed, 11 Mar 2009 09:30:59 GMT
I made more test. I think my guess may be the cause.
In a latest test, there are 11 killed map tasks, and there are also 11
expired leases.

I think may be the MapReduce code should be improved.

Schubert
On Wed, Mar 11, 2009 at 4:38 PM, schubert zhang <zsongbo@gmail.com> wrote:

> Hi all,
> I am running a MapReduce job to read regions(row ranges) in Map task. The
> job can work completely and correctly.
>
> But, usually, I find following INFO in hbase regionservre's log file.
>
> 2009-03-11 15:57:06,428 INFO
> org.apache.hadoop.hbase.regionserver.HRegionServer: Scanner
> -6710483098243538624 lease expired
>
> To debug this issue, I add some new log information in:
> src/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
> (1) in addScanner(), I add:
>  LOG.info("----Schubert Add Scanner " + scannerName + " lease create!");
> (2) in close(), I add:
> LOG.info("----Schubert close " + scannerName + " lease cancel!");
>
> Then I redo my mapreduce job. And I found some leases/scanners are not
> closed.
>
> I can make sure that my mapreduce RecordReader implements the close()
> method and calls the close() method of the scanner.
>
> following log info is copy from one region server of my HBase clasuer, and
> to make clear, I move the Add and Close together.
>
> 2009-03-11 15:52:41,406 INFO
> org.apache.hadoop.hbase.regionserver.HRegionServer: ----Schubert Add Scanner
> 5785283094618320498 lease create!
> 2009-03-11 15:54:08,423 INFO
> org.apache.hadoop.hbase.regionserver.HRegionServer: ----Schubert close
> 5785283094618320498 lease cancel!
>
> 2009-03-11 15:52:46,493 INFO
> org.apache.hadoop.hbase.regionserver.HRegionServer: ----Schubert Add Scanner
> -3726092592914221244 lease create!
> 2009-03-11 15:54:11,676 INFO
> org.apache.hadoop.hbase.regionserver.HRegionServer: ----Schubert close
> -3726092592914221244 lease cancel!
>
> 2009-03-11 15:52:51,984 INFO
> org.apache.hadoop.hbase.regionserver.HRegionServer: ----Schubert Add Scanner
> 313013793996434532 lease create!
> 2009-03-11 15:54:18,428 INFO
> org.apache.hadoop.hbase.regionserver.HRegionServer: ----Schubert close
> 313013793996434532 lease cancel!
>
> 2009-03-11 15:54:10,213 INFO
> org.apache.hadoop.hbase.regionserver.HRegionServer: ----Schubert Add Scanner
> -5389407006476765661 lease create!
> 2009-03-11 15:55:48,337 INFO
> org.apache.hadoop.hbase.regionserver.HRegionServer: ----Schubert close
> -5389407006476765661 lease cancel!
>
> 2009-03-11 15:54:13,505 INFO
> org.apache.hadoop.hbase.regionserver.HRegionServer: ----Schubert Add Scanner
> 8463558669492796494 lease create!
> 2009-03-11 15:55:59,152 INFO
> org.apache.hadoop.hbase.regionserver.HRegionServer: ----Schubert close
> 8463558669492796494 lease cancel!
>
> 2009-03-11 15:54:20,118 INFO
> org.apache.hadoop.hbase.regionserver.HRegionServer: ----Schubert Add Scanner
> 8149590914348186380 lease create!
> 2009-03-11 15:56:04,350 INFO
> org.apache.hadoop.hbase.regionserver.HRegionServer: ----Schubert close
> 8149590914348186380 lease cancel!
>
> 2009-03-11 15:54:29,466 INFO
> org.apache.hadoop.hbase.regionserver.HRegionServer: ----Schubert Add Scanner
> 4137760362936278647 lease create!
> 2009-03-11 15:56:10,765 INFO
> org.apache.hadoop.hbase.regionserver.HRegionServer: ----Schubert close
> 4137760362936278647 lease cancel!
>
> 2009-03-11 15:55:34,786 INFO
> org.apache.hadoop.hbase.regionserver.HRegionServer: ----Schubert Add Scanner
> -5919619533272076177 lease create!
> 2009-03-11 15:56:50,472 INFO
> org.apache.hadoop.hbase.regionserver.HRegionServer: Scanner
> -5919619533272076177 lease expired
>
> 2009-03-11 15:55:36,961 INFO
> org.apache.hadoop.hbase.regionserver.HRegionServer: ----Schubert Add Scanner
> -7919803182075534481 lease create!
> 2009-03-11 15:57:04,200 INFO
> org.apache.hadoop.hbase.regionserver.HRegionServer: Scanner
> -7919803182075534481 lease expired
>
> 2009-03-11 15:55:45,992 INFO
> org.apache.hadoop.hbase.regionserver.HRegionServer: ----Schubert Add Scanner
> 2812856283312715383 lease create!
> 2009-03-11 15:57:14,442 INFO
> org.apache.hadoop.hbase.regionserver.HRegionServer: Scanner
> 2812856283312715383 lease expired
>
> 2009-03-11 15:56:03,033 INFO
> org.apache.hadoop.hbase.regionserver.HRegionServer: ----Schubert Add Scanner
> -6710483098243538624 lease create!
> 2009-03-11 15:57:06,428 INFO
> org.apache.hadoop.hbase.regionserver.HRegionServer: Scanner
> -6710483098243538624 lease expired
>
> Wen can see, the
> Scanners  -5919619533272076177, -7919803182075534481, 2812856283312715383
> and -6710483098243538624 are not closed.
>
> I guess, it maybe caused by MapReduce kill task (speculative).
>
> Schubert
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message