hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From baggio liu <baggi...@gmail.com>
Subject Re: Scan cause too many connection
Date Fri, 30 Jul 2010 05:52:09 GMT
Hi Angus,
   2. During scanning, it'll new StoreFile one by one. And in constructor of
StoreFile , HFile.Reader will be created. HFile.Reader act as DFSClient,
it'll keep a connection with DataNode when something should be read.

   1.After checking code, and I've seen in scanner close method,
HFile.Reader has not been closed. Is that used for reuse HFile ?
      I try to close the scanner explictly, but it cause meta region can not
be loaded when start up. Both get() and scan()  use the same scanner...

Thanks for your reply.

Thanks & Best regards
LiuJinglong

在 2010年7月30日 上午11:33,Angus He <angushe@gmail.com>写道:

> 1. try to close the scanner explictly?
>
> 2. I do not think HBase will issue a new connection for each StoreFile
> for the scan operation.
>
> 2010/7/29 baggio liu <baggioss@gmail.com>:
> > Hi all,
> >    We have 53 machines in our hbase cluster and run 6 clients to scan a
> > table. During scanning, we found when a region is scanning , it'll new
> > StoreFile object, create a connection to datanode (in fact , create HFile
> > Reader), so the number of connection increases by the number of store
> files.
> > We have many store files ,(cross several regions and has not reach minor
> > compaction thredshould), too many connections has been created. And after
> > scanning, the connection will not closed.
> >    As the result of it , a machine which act as region server has too
> high
> > system CPU, and hung for a long time.
> >    My question is :
> >   1. Why we don't close connection ( in fact, we don't close
> > HFile.Reader)after we complete to scan table ? We wanna to reuse
> connection
> > in the next scan ??
> >   2. How can we limit the connection number ?
> >
> > Thanks & Best regards
> > LiuJinglong
> >
>
>
>
> --
> Regards
> Angus
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message