hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Angus He <angu...@gmail.com>
Subject Re: Scan cause too many connection
Date Fri, 30 Jul 2010 06:51:25 GMT
Hi Baggio,


>   2. During scanning, it'll new StoreFile one by one. And in constructor of
> StoreFile , HFile.Reader will be created. HFile.Reader act as DFSClient,
> it'll keep a connection with DataNode when something should be read.

Yes, There is a HFile.Reader opend in StoreFile constructor.
But I am afraid there is only one instance in HBase for each
StoreFile. And I just confirmed this
in my dev machine. (HBase running in standalone mode). After many scan
operations, still only one
instance for each StoreFile.

>   1.After checking code, and I've seen in scanner close method,
> HFile.Reader has not been closed. Is that used for reuse HFile ?
>      I try to close the scanner explictly, but it cause meta region can not
> be loaded when start up. Both get() and scan()  use the same scanner...

It seems there is only one HFile.Reader for each StoreFile, and the
HFile.Reader instance is shared by multiple scannners.

As for closing the scannner explicitly, it works all right in our
case.  And really do not know what happened in your case.

By the way, how do you get the scanner,by HTable or other ways?


>
> Thanks & Best regards
> LiuJinglong
>
> 在 2010年7月30日 上午11:33,Angus He <angushe@gmail.com>写道:
>
>> 1. try to close the scanner explictly?
>>
>> 2. I do not think HBase will issue a new connection for each StoreFile
>> for the scan operation.
>>
>> 2010/7/29 baggio liu <baggioss@gmail.com>:
>> > Hi all,
>> >    We have 53 machines in our hbase cluster and run 6 clients to scan a
>> > table. During scanning, we found when a region is scanning , it'll new
>> > StoreFile object, create a connection to datanode (in fact , create HFile
>> > Reader), so the number of connection increases by the number of store
>> files.
>> > We have many store files ,(cross several regions and has not reach minor
>> > compaction thredshould), too many connections has been created. And after
>> > scanning, the connection will not closed.
>> >    As the result of it , a machine which act as region server has too
>> high
>> > system CPU, and hung for a long time.
>> >    My question is :
>> >   1. Why we don't close connection ( in fact, we don't close
>> > HFile.Reader)after we complete to scan table ? We wanna to reuse
>> connection
>> > in the next scan ??
>> >   2. How can we limit the connection number ?
>> >
>> > Thanks & Best regards
>> > LiuJinglong
>> >
>>
>>
>>
>> --
>> Regards
>> Angus
>>
>



-- 
Regards
Angus

Mime
View raw message