kudu-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Todd Lipcon <t...@cloudera.com>
Subject Re: Performance Question
Date Sat, 28 May 2016 04:19:19 GMT
On Fri, May 27, 2016 at 8:20 PM, Benjamin Kim <bbuild11@gmail.com> wrote:

> Hi Mike,
>
> First of all, thanks for the link. It looks like an interesting read. I
> checked that Aerospike is currently at version 3.8.2.3, and in the article,
> they are evaluating version 3.5.4. The main thing that impressed me was
> their claim that they can beat Cassandra and HBase by 8x for writing and
> 25x for reading. Their big claim to fame is that Aerospike can write 1M
> records per second with only 50 nodes. I wanted to see if this is real.
>

1M records per second on 50 nodes is pretty doable by Kudu as well,
depending on the size of your records and the insertion order. I've been
playing with a ~70 node cluster recently and seen 1M+ writes/second
sustained, and bursting above 4M. These are 1KB rows with 11 columns, and
with pretty old HDD-only nodes. I think newer flash-based nodes could do
better.


>
> To answer your questions, we have a DMP with user profiles with many
> attributes. We create segmentation information off of these attributes to
> classify them. Then, we can target advertising appropriately for our sales
> department. Much of the data processing is for applying models on all or if
> not most of every profile’s attributes to find similarities (nearest
> neighbor/clustering) over a large number of rows when batch processing or a
> small subset of rows for quick online scoring. So, our use case is a
> typical advanced analytics scenario. We have tried HBase, but it doesn’t
> work well for these types of analytics.
>
> I read, that Aerospike in the release notes, they did do many improvements
> for batch and scan operations.
>
> I wonder what your thoughts are for using Kudu for this.
>

Sounds like a good Kudu use case to me. I've heard great things about
Aerospike for the low latency random access portion, but I've also heard
that it's _very_ expensive, and not particularly suited to the columnar
scan workload. Lastly, I think the Apache license of Kudu is much more
appealing than the AGPL3 used by Aerospike. But, that's not really a direct
answer to the performance question :)


>
> Thanks,
> Ben
>
>
> On May 27, 2016, at 6:21 PM, Mike Percy <mpercy@cloudera.com> wrote:
>
> Have you considered whether you have a scan heavy or a random access heavy
> workload? Have you considered whether you always access / update a whole
> row vs only a partial row? Kudu is a column store so has some
> awesome performance characteristics when you are doing a lot of scanning of
> just a couple of columns.
>
> I don't know the answer to your question but if your concern is
> performance then I would be interested in seeing comparisons from a perf
> perspective on certain workloads.
>
> Finally, a year ago Aerospike did quite poorly in a Jepsen test:
> https://aphyr.com/posts/324-jepsen-aerospike
>
> I wonder if they have addressed any of those issues.
>
> Mike
>
> On Friday, May 27, 2016, Benjamin Kim <bbuild11@gmail.com> wrote:
>
>> I am just curious. How will Kudu compare with Aerospike (
>> http://www.aerospike.com)? I went to a Spark Roadshow and found out
>> about this piece of software. It appears to fit our use case perfectly
>> since we are an ad-tech company trying to leverage our user profiles data.
>> Plus, it already has a Spark connector and has a SQL-like client. The
>> tables can be accessed using Spark SQL DataFrames and, also, made into SQL
>> tables for direct use with Spark SQL ODBC/JDBC Thriftserver. I see from the
>> work done here http://gerrit.cloudera.org:8080/#/c/2992/ that the Spark
>> integration is well underway and, from the looks of it lately, almost
>> complete. I would prefer to use Kudu since we are already a Cloudera shop,
>> and Kudu is easy to deploy and configure using Cloudera Manager. I also
>> hope that some of Aerospike’s speed optimization techniques can make it
>> into Kudu in the future, if they have not been already thought of or
>> included.
>>
>> Just some thoughts…
>>
>> Cheers,
>> Ben
>
>
>
> --
> --
> Mike Percy
> Software Engineer, Cloudera
>
>
>
>


-- 
Todd Lipcon
Software Engineer, Cloudera

Mime
View raw message