hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Stack <st...@duboce.net>
Subject Re: multiple scanners on same table/region
Date Tue, 27 Apr 2010 13:31:16 GMT
You may be running into HBASE-2481?
St.Ack

On Tue, Apr 27, 2010 at 1:30 AM, steven zhuang
<steven.zhuang.1984@gmail.com> wrote:
> the first thread can be found at:
> http://permalink.gmane.org/gmane.comp.java.hadoop.hbase.user/10074
>
>           After some dig, it seems that the problem is caused by long pause
> between two "scanner.next()" call.
>
>           In my case the program has to spent a relatively long while to
> process one row, when it calls "scanner.next()" again, seems that the
> returned Result will be null even if there should be more rows in the
> tables.  The rowcaching is set to 1.
>            I have checked some of the source code, seems there is some
> mechanism which will call the close() method of the ClientScanner, but I am
> still checking.
>           I don't know if there is a certain timeout on
> ClientScanner/ScannerCallable after a row has been successfully returned,
> seems that timeout cause  my problem here.
>
>          Any reply is appreciated.
>
>
> On Fri, Apr 23, 2010 at 11:10 AM, steven zhuang <
> steven.zhuang.1984@gmail.com> wrote:
>
>> hi,
>>           sorry I start another thread here.  This mail is actually answer
>> to a previous thread "multiple scanners on same table will cause problem?
>> Scan results change among different tries.".
>>           the mail system kept saying that I am spamming, now it seems that
>> it's right! :)
>>
>> here is my reply to people in that thread:
>>
>>       I don't know if there is a limit on reads to a single row/region in
>> HBase, but if there is, I might have exceeded that limit.   :(
>>       in my case, there are hundreds of rows, with dozens of kilos of cells
>> in a row(a 256 MB region may contain 10- rows). for each row, I started a
>> thread on each CF, there are 8 of them, so there might be dozens of scanners
>> on the same region.
>>       and, to Tim, I could not see your attached mail, my test code is
>> pasted below, it just iterate on the rows and column families, output all
>> the cells.
>>
>>   private void doScan() throws Exception {
>>  if (null == CopyOfTestTTT234.table) {
>> return;
>> }
>> Scan s = new Scan();
>>  s.setStartRow("aaa".getBytes());
>> s.setStopRow("ccc".getBytes());
>> s.setCaching(CopyOfTestTTT234.ROWCACHING);  //it's 1 here.
>>  ResultScanner scanner = CopyOfTestTTT234.table.getScanner(s);
>> while (true) {
>> Result row = scanner.next();
>>  if(null==row) break;
>> String rowKey = new String(row.getRow());
>> NavigableMap<byte[], NavigableMap<byte[], byte[]>> fm = row
>>  .getNoVersionMap();
>> while (fm.size() > 0) {
>> Entry<byte[], NavigableMap<byte[], byte[]>> ee = fm
>>  .pollFirstEntry();
>> String fName = new String(ee.getKey());
>> NavigableMap<byte[], byte[]> ff = ee.getValue();
>>  while (ff.size() > 0) {
>> Entry<byte[], byte[]> cell = ff.pollFirstEntry();
>> String key = new String(cell.getKey());
>>  String val = new String(cell.getValue());
>> System.out.println(Thread.currentThread().hashCode() + "\t"
>>  + rowKey + "\t" + fName + "\t" + key + "\t" + val);
>> }
>> }
>>  }
>> }
>>
>

Mime
View raw message