hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Thomas Downing <tdown...@proteus-technologies.com>
Subject HBase at high ingest rates
Date Fri, 13 Aug 2010 09:37:25 GMT
First, I want to thank all on this list who responded to all
my questions, and all the HBase developers as well.

After trying both hadoop-0.20-append and Cloudera's
latest, along with various settings, we are still leaking both
file and socket handles at high ingest rates.  The handle
usage increases slowly, but linearly, till a process or OS
limit is reached.  The larger problem is that HBase does
not recover from that situation.  In fact, if once the file
handles were driven high, but less than quota, and the
ingest was slackened significantly, there was no recovery.
If ingest was completely stopped and restarted, there
was recovery.

By high ingest rate I mean about 1200 records/sec per
node, 2K per record, 50 records per batch put.

I truly wish I had the time to dig in to find the answers, or
at least indications, but my time is not my own.  We will
keeping a close eye on HBase, and will likely revisit it
before long.

Thanks again
Thomas Downing

View raw message