hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Tatsuya Kawano <tatsuy...@snowcocoa.info>
Subject Re: random write slower than random read in HBase 0.20.0
Date Sat, 24 Oct 2009 09:56:03 GMT
Hi Jun,

On Sat, Oct 24, 2009 at 10:56 AM, Jun Li wrote:
> I have a cluster of 13 Linux RedHat machines (each one with 4 CPU
> cores and 8GB RAM) and I configured it to form a HBase cluster of 1
> master and 12 region servers, with HBase-0.20.0 code base.


> As a comparison, in the measurement result reported by
> Schubert Zhang (in the article mentioned above),  with the smaller
> test environment that he had (1 master and 4 slaves, 4 CPU
> core/machine, 8GB RAM per machine),  it was reported to obtain the
> random write of 11366 row/second and the random read of 4424
> row/second. That is, the random read does get significantly improved
> in my case as I had more machines in the cluster, but not the random
> write.
>
> Please help on making comments and suggestions, for possible
> performance improvement on random write.


Did you also run the same test with 4 region servers or some smaller
numbers than 12?

Your assumption is based on both random reads and random writes would
scale at the same degree as you add more region servers, but they
probably not. HBase employs a block cache for reads, and this could
make random reads to increase the performance quicker than random
writes.

If you have more region servers, you have more RAM for block cache.
This makes each region server to read blocks from the disk less often,
and this could dramatically improve the random reads performance. But,
on the other hand, a write operation always involves a disk access to
append WAL on the disk. This will make random writes slower to
increase the performance than random reads.

So, if you run the same test with smaller number of region servers. It
could yield a similar performance to the "random reads" plot in the
BigTable paper. And as you add more region servers, the block cache
will become more effective and the performance could be getting closer
to the "random reads (mem)" plot.

Well, I'm not totally sure if I'm right; I've never had a chance to
run a large HBase cluster. So, if you run your test on a smaller and
larger clusters and share us your results, that would be great.


As for improving write performance, I would suggest
“setAutoFlush(false)”, “setWriteBufferSize(**)” so that you can put
multiple records in a single flushCommits(). This will give you
instant performance boost. You could do this by updating your web
service to accept multiple puts in a single RPC.

There should other HBase parameters to optimize, but I'm not the
expert who can tell you how to optimize them for your environment. And
I also think it would be nice to check how random read improves the
performance in different sizes of clusters **before** going crazy to
tweak those parameters.

Thanks,

-- 
Tatsuya Kawano (Mr.)
Tokyo, Japan



On Sat, Oct 24, 2009 at 10:56 AM, Jun Li wrote:
> I have a cluster of 13 Linux RedHat machines (each one with 4 CPU
> cores and 8GB RAM) and I configured it to form a HBase cluster of 1
> master and 12 region servers, with HBase-0.20.0 code base.  For the
> cluster configuration, I followed what was described in the article
> “HBase-0.20.0 Performance Evaluation” by Anty Rao and Schubert Zhang
> on August 21, 2009 (accessible from:
> http://www.slideshare.net/schubertzhang/hbase-0200-performance-evaluation,
> which was also discussed in one of the HBase  mailing list archives in
> the last several months). So I allocated 4GB heap for each
> HBase-related process, with the “hbase.regionserver.handler.count” set
> to be “20”.  A small difference is that I used the HBase managed
> Zoo-keeper to manage the Hbase Masters.
>
> I then set up an Hbase table with a row key of 48 bytes, and a column
> that holds about 20 Bytes data.  For a single client, I was able to
> get in average, the write of 0.6 milliseconds per row (random write),
> and the read of 0.4 milliseconds per row (random read).
>
> Then  I  had each machine in the cluster to launch 1, or 2,  or 3
> client test applications, with each client test application read/write
> 100000 rows for each test run, for throughput testing.  From my
> measurement results, I found that the random write will have best
> measured performance when each machine having 2 clients (totally
> 2*13=26 clients in the cluster), with 8500 rows/second; and the random
> read will have almost the same throughput for 2 or 3 clients, with
> 35000 rows/second.
>
> Since I designed the Hbase table to be accessed via the Web Service
> for random read or write data access, the HTable instance is created
> in a default mode, that is, I did not have customized settings for
> “setAutoFlush(false)”, “setWriteBufferSize(**)”, etc.
>
> I also incorporated the HTablePool so that each test client
> application will use the same HTable instance. But I observed little
> improvement on my random write testing from the above numbers, with or
> without table pooling.
>
> So the question that I have is that, following the original Google’s
> BigTable paper, should Random Write be always much faster than Random
> Read?   If that is the case, what are the tunable parameters in terms
> of HBase setup that I can explore to improve the Random Write speed.
>
> I also downloaded the PerformanceEvaluation.java that is patched by
> Schubert Zhang (the link to the code is in the article mentioned
> above), and used it to test my cluster’s performance as well. In my
> cluster, to read/write 4,194,280 rows, I will need 274 seconds for
> random writes (translated to15307 row/second) and 305 seconds for
> random reads (translated to 13751 row/second). Notice that although
> random write is still faster than random read, but they are almost
> compatible.  As a comparison, in the measurement result reported by
> Schubert Zhang (in the article mentioned above),  with the smaller
> test environment that he had (1 master and 4 slaves, 4 CPU
> core/machine, 8GB RAM per machine),  it was reported to obtain the
> random write of 11366 row/second and the random read of 4424
> row/second. That is, the random read does get significantly improved
> in my case as I had more machines in the cluster, but not the random
> write.
>
> Please help on making comments and suggestions, for possible
> performance improvement on random write.
>
> Regards,
>
>
> Jun Li
>

Mime
View raw message