what is your hbase version?
the close_wait is in regionserver process, probably rpc connection close()
is not call due to some reason? can you pastebin the related regionserver
log?
On Thu, Jul 24, 2014 at 3:16 PM, ramkrishna vasudevan <
ramkrishna.s.vasudevan@gmail.com> wrote:
> Sorry about that.
> https://issues.apache.org/jira/browse/HBASE-9393. Similar issue has been
> reported.
>
> What version of HBase and HDFS are you using? See if things are similar.
>
> Regards
> Ram
>
>
> On Thu, Jul 24, 2014 at 12:33 PM, Shankar hiremath <
> shankar.hiremath@huawei.com> wrote:
>
> > Hi Ram,
> >
> > Actually It is not file FD, it is socket FD which is in CLOSE_WAIT state
> > (tcp socket connection)
> >
> > > java 18028 shankar1 194u IPv6 18548560 0t0 TCP
> > > XX-XX-XX-XX:62675->XX-XX-XX-XX:busboy (CLOSE_WAIT)
> >
> > Regards
> > -Shankar
> >
> >
> > This e-mail and its attachments contain confidential information from
> > HUAWEI, which is intended only for the person or entity whose address is
> > listed above. Any use of the information contained herein in any way
> > (including, but not limited to, total or partial disclosure,
> reproduction,
> > or dissemination) by persons other than the intended recipient(s) is
> > prohibited. If you receive this e-mail in error, please notify the sender
> > by phone or email immediately and delete it!
> >
> >
> >
> >
> > -----Original Message-----
> > From: ramkrishna vasudevan [mailto:ramkrishna.s.vasudevan@gmail.com]
> > Sent: 24 July 2014 PM 12:23
> > To: user@hbase.apache.org
> > Subject: Re: RegionServer many socket fds are in CLOSE_WAIT and not
> > getting cleared
> >
> > Does your regions have store files in them?
> > In HBase all the store files are opened and the readers are kept open
> > always when the region opening happens. When there is a compaction
> > happening the count of those file handlers open gets reduced based on the
> > number of files compacted.
> >
> > Regards
> > Ram
> >
> >
> > On Thu, Jul 24, 2014 at 12:19 PM, Shankar hiremath <
> > shankar.hiremath@huawei.com> wrote:
> >
> > > Dear All,
> > >
> > > Observation:
> > > I have HBase cluster with Kerberos enabled, when the Region Server
> > > startups then we observed some 20-30 socket fds are in CLOSE_WAIT
> > > state, After that when the Region Server starts opening then the
> > > socket fds in CLOSE_WAIT starts increasing gradually (almost same as
> > > number of regions opened by the region server) And all these
> > > CLOSE_WAIT states are not getting cleared up,
> > >
> > > /hbase> lsof -i | grep `jps | grep RegionServer | cut -d " " -f1`
> > > java 18028 shankar1 118u IPv6 18552894 0t0 TCP
> > > XX-XX-XX-XX:60020 (LISTEN)
> > > java 18028 shankar1 160u IPv6 18548520 0t0 TCP *:60030
> > (LISTEN)
> > > java 18028 shankar1 167u IPv6 18548522 0t0 TCP
> > > XX-XX-XX-XX:42534-> XX-XX-XX-XX:eforward (ESTABLISHED)
> > > java 18028 shankar1 172u IPv6 18552916 0t0 TCP
> > > XX-XX-XX-XX:42535-> XX-XX-XX-XX:eforward (ESTABLISHED)
> > > java 18028 shankar1 173u IPv6 18551227 0t0 TCP
> > > XX-XX-XX-XX:49646->XX-XX-XX-XX:60000 (ESTABLISHED)
> > > java 18028 shankar1 178u IPv6 18551237 0t0 TCP
> > > XX-XX-XX-XX:62668->XX-XX-XX-XX:busboy (ESTABLISHED)
> > > java 18028 shankar1 185u IPv6 18548549 0t0 TCP
> > > XX-XX-XX-XX:21856-> XX-XX-XX-XX:eforward (ESTABLISHED)
> > > java 18028 shankar1 187u IPv6 18548558 0t0 TCP
> > > XX-XX-XX-XX:62673->XX-XX-XX-XX:busboy (ESTABLISHED)
> > > java 18028 shankar1 188u IPv6 18601323 0t0 TCP
> > > XX-XX-XX-XX:63168->XX-XX-XX-XX:busboy (CLOSE_WAIT)
> > > java 18028 shankar1 189u IPv6 18601322 0t0 TCP
> > > XX-XX-XX-XX:63167->XX-XX-XX-XX:busboy (CLOSE_WAIT)
> > > java 18028 shankar1 190u IPv6 18601324 0t0 TCP
> > > XX-XX-XX-XX:63169->XX-XX-XX-XX:busboy (CLOSE_WAIT)
> > > java 18028 shankar1 191r IPv6 18592423 0t0 TCP
> > > XX-XX-XX-XX:63087->XX-XX-XX-XX:busboy (CLOSE_WAIT)
> > > java 18028 shankar1 193u IPv6 18593210 0t0 TCP
> > > XX-XX-XX-XX:63090->XX-XX-XX-XX:busboy (CLOSE_WAIT)
> > > java 18028 shankar1 194u IPv6 18548560 0t0 TCP
> > > XX-XX-XX-XX:62675->XX-XX-XX-XX:busboy (CLOSE_WAIT)
> > > java 18028 shankar1 195u IPv6 18592428 0t0 TCP
> > > XX-XX-XX-XX:63093->XX-XX-XX-XX:busboy (CLOSE_WAIT)
> > > java 18028 shankar1 196u IPv6 18593218 0t0 TCP
> > > XX-XX-XX-XX:63096->XX-XX-XX-XX:busboy (CLOSE_WAIT)
> > > java 18028 shankar1 197u IPv6 18591423 0t0 TCP
> > > XX-XX-XX-XX:63105->XX-XX-XX-XX:busboy (CLOSE_WAIT)
> > > java 18028 shankar1 201u IPv6 18592431 0t0 TCP
> > > XX-XX-XX-XX:63099->XX-XX-XX-XX:busboy (CLOSE_WAIT)
> > > java 18028 shankar1 202u IPv6 18592433 0t0 TCP
> > > XX-XX-XX-XX:63102->XX-XX-XX-XX:busboy (CLOSE_WAIT)
> > > java 18028 shankar1 203u IPv6 18552317 0t0 TCP
> > > XX-XX-XX-XX:62681->XX-XX-XX-XX:busboy (CLOSE_WAIT)
> > >
> > >
> >
> ............................................................................................................
> > >
> > >
> >
> ............................................................................................................
> > >
> > >
> >
> ............................................................................................................
> > >
> > >
> >
> ............................................................................................................
> > >
> > >
> >
> ............................................................................................................
> > >
> > > any input or suggestion will be helpful, or is it a bug.
> > >
> > > Regards
> > > -Shankar
> > >
> > >
> > > [X]
> > > This e-mail and its attachments contain confidential information from
> > > HUAWEI, which is intended only for the person or entity whose address
> > > is listed above. Any use of the information contained herein in any
> > > way (including, but not limited to, total or partial disclosure,
> > > reproduction, or dissemination) by persons other than the intended
> > > recipient(s) is prohibited. If you receive this e-mail in error,
> > > please notify the sender by phone or email immediately and delete it!
> > > [X]
> > >
> > >
> > >
> > >
> > >
> > >
> >
>
|