hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "M.Deniz OKTAR" <deniz.ok...@gmail.com>
Subject Re: VMWare and Hadoop/Hbase
Date Sat, 14 May 2011 02:04:28 GMT
Thanks david :)

--
M.Deniz OKTAR

iletken Recommendation Technologies
http://www.iletken.com.tr
Tel:      +90(212)328-0290
GSM:   +90(533)477-6358



On Sat, May 14, 2011 at 12:39 AM, Buttler, David <buttler1@llnl.gov> wrote:

> This is a fantastic test and should be made more public.  Great work
> Dave
>
>
> -----Original Message-----
> From: M.Deniz OKTAR [mailto:deniz.oktar@gmail.com]
> Sent: Friday, May 13, 2011 2:20 PM
> To: user@hbase.apache.org; apurtell@apache.org
> Subject: Re: VMWare and Hadoop/Hbase
>
> I'v tried this in our internal tests with Xen,
>
> We tried to see if the performance degrades consistently with the amount of
> resources we take away from the standalone machine to the visualized one.
>
> The cluster I tested is below, took one disk  (25% of 4) ,1 core with
> medium
> priority (12,5% of 8) and ~1100 mb of memory (10%) to the second
> virtual machine and the results of the virtual machines with the
> remaining resources were awful . Writing performance degraded around 25%,
> which is cool. But the read was around 3x slower which pointed out that
> Hypervisor is not so good at handling huge random disk accesses.
>
> Tests: Yahoo benchmark  100M
>
> Cluster:
> 5x cluster (1 namenode 4 data nodes)
> xeon 5620
> 12gb ram
> 4x sata 7200 rpm drives
>
> Results:
>
> === TEST 1: 100M inserts ===
>
> ---STANDALONE CLUSTER---
>
> YCSB Client 0.1
> Command line: -db com.yahoo.ycsb.db.HBaseClient -load -p
> columnfamily=values
> -P myworkloads/4 -threads 100 -s
> [OVERALL], RunTime(ms), 5374309.0
> [OVERALL], Throughput(ops/sec), 18607.043249653118
> [INSERT], Operations, 100000000
> [INSERT], AverageLatency(ms), 4.88454509
> [INSERT], MinLatency(ms), 0
> [INSERT], MaxLatency(ms), 92837
>
>
>
>
> ---VIRTUAL CLUSTER---
>
> Command line: -db com.yahoo.ycsb.db.HBaseClient -load -p
> columnfamily=values
> -P
> myworkloads/4 -threads 100 -s
> [OVERALL], RunTime(ms), 6912776.0
> [OVERALL], Throughput(ops/sec), 14465.968519737946
> [INSERT], Operations, 100000000
> [INSERT], AverageLatency(ms), 6.36836144
> [INSERT], MinLatency(ms), 0
> [INSERT], MaxLatency(ms), 104990
>
>
> === TEST2: 100M read/update/write (transaction) ===
>
>
>
> ---STANDALONE CLUSTER---
>
> YCSB Client 0.1
> Command line: -db com.yahoo.ycsb.db.HBaseClient -p columnfamily=values -P
> myworkloads/4 -threads 100 -s
> [OVERALL], RunTime(ms), 9012970.0
> [OVERALL], Throughput(ops/sec), 3328.53654233843
> [UPDATE], Operations, 4500829
> [UPDATE], AverageLatency(ms), 0.09758402285445637
> [UPDATE], MinLatency(ms), 0
> [UPDATE], MaxLatency(ms), 5477
>
>
>
> ---VIRTUAL CLUSTER---
>
> YCSB Client 0.1
> Command line: -db com.yahoo.ycsb.db.HBaseClient -p columnfamily=values -P
> myworkloads/4 -threads 100 -s
> [OVERALL], RunTime(ms), 2.0272831E7
> [OVERALL], Throughput(ops/sec), 1479.813056203152
> [UPDATE], Operations, 4502501
> [UPDATE], AverageLatency(ms), 2.803586717693122
> [UPDATE], MinLatency(ms), 0
> [UPDATE], MaxLatency(ms), 560605
>
> --
> M.Deniz OKTAR
>
> iletken Recommendation Technologies
> http://www.iletken.com.tr
> Tel:      +90(212)328-0290
> GSM:   +90(533)477-6358
>
>
>
> On Mon, May 9, 2011 at 10:41 PM, Andrew Purtell <apurtell@apache.org>
> wrote:
>
> > It is not advisable to do this.
> >
> > Hadoop/HBase is very I/O intensive. They should have dedicated hardware.
> > Why add the overhead of Hypervisor mediation on the I/O path then?
> >
> > --- On Mon, 5/9/11, Vishal Kapoor <vishal.kapoor.in@gmail.com> wrote:
> >
> > > From: Vishal Kapoor <vishal.kapoor.in@gmail.com>
> > > Subject: VMWare and Hadoop/Hbase
> > > To: user@hbase.apache.org
> > > Date: Monday, May 9, 2011, 12:24 PM
> > > We were wondering if its advisable to
> > > provision hbase/hadoop nodes as VMWare
> > > instances?
> > > any suggestions?
> > >
> > > thanks,
> > > Vishal
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message