kudu-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Todd Lipcon <t...@cloudera.com>
Subject Re: Low ingestion rate from Kafka
Date Tue, 31 Oct 2017 05:39:00 GMT
Hey Chao,

Nice to hear you are checking out Kudu.

What are you using to consume from Kafka and write to Kudu? Is it possible
that it is Java code and you are using the SYNC flush mode? That would
result in a separate round trip for each record and thus very low
throughput.

Todd

On Oct 30, 2017 10:23 PM, "Chao Sun" <sunchao@uber.com> wrote:

Hi,

We are evaluating Kudu (version kudu 1.3.0-cdh5.11.1, revision
af02f3ea6d9a1807dcac0ec75bfbca79a01a5cab) on a 8-node cluster.
The data are coming from Kafka at a rate of around 30K / sec, and hash
partitioned into 128 buckets. However, with default settings, Kudu can only
consume the topics at a rate of around 1.5K / second. This is a direct
ingest with no transformation on the data.

Could this because I was using the default configurations? also we are
using Kudu on HDD - could that also be related?

Any help would be appreciated. Thanks.

Best,
Chao

Mime
View raw message