kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Emmanuel <ele...@msn.com>
Subject RE: Post on running Kafka at LinkedIn
Date Fri, 20 Mar 2015 22:41:05 GMT
This is why I'm confused because I'm tryign to benchmark and I see numbers that seem pretty
low to me...8000 events/sec on 2 brokers with 3CPU each and 5 partitions should be way faster
than this and I don't know where to start to debug...
the kafka-consumer-perf-test script gives me ridiculously low numbers (1000 events/sec/thread)

So what could be causing this?
From: jbringhurst@linkedin.com.INVALID
To: users@kafka.apache.org
Subject: Re: Post on running Kafka at LinkedIn
Date: Fri, 20 Mar 2015 22:16:29 +0000

Keep in mind that these brokers aren't really stressed too much at any given time -- we need
to stay ahead of the capacity curve.
Your message throughput will really just depend on what hardware you're using. However, in
the past, we've benchmarked at 400,000 to more than 800,000 messages / broker / sec, depending
on configuration (https://engineering.linkedin.com/kafka/benchmarking-apache-kafka-2-million-writes-second-three-cheap-machines).

On Mar 20, 2015, at 3:03 PM, Emmanuel <eleroy@msn.com> wrote:800B messages / day = 9.26M
messages / sec over 1100 brokers 
= ~8400 message / broker / sec
Do I get this right?
Trying to benchmark my own test cluster and that's what I see with 2 brokers...Just wondering
if my numbers are good or bad...

Subject: Re: Post on running Kafka at LinkedIn
From: clark@kafka.guru
Date: Fri, 20 Mar 2015 14:27:58 -0700
To: users@kafka.apache.org

Yep! We are growing :)


Sent from my iPhone

On Mar 20, 2015, at 2:14 PM, James Cheng <jcheng@tivo.com> wrote:

Amazing growth numbers.

At the meetup on 1/27, Clark Haskins presented their Kafka usage at the time. It was:

Bytes in: 120 TB
Messages In: 585 million
Bytes out: 540 TB
Total brokers: 704

In Todd's post, the current numbers:

Bytes in: 175 TB (45% growth)
Messages In: 800 billion (36% growth)
Bytes out: 650 TB (20% growth)
Total brokers: 1100 (56% growth)

That much growth in just 2 months? Wowzers.


On Mar 20, 2015, at 11:30 AM, James Cheng <jcheng@tivo.com> wrote:

For those who missed it:

The Kafka Audit tool was also presented at the 1/27 Kafka meetup:

Recorded video is here, starting around the 40 minute mark:

Slides are here:


On Mar 20, 2015, at 9:47 AM, Todd Palino <tpalino@gmail.com> wrote:

For those who are interested in detail on how we've got Kafka set up at
LinkedIn, I have just published a new posted to our Engineering blog titled
"Running Kafka at Scale"


It's a general overview of our current Kafka install, tiered architecture,
audit, and the libraries we use for producers and consumers. You'll also be
seeing more posts from the SRE team here in the coming weeks on deeper
looks into both Kafka and Samza.

Additionally, I'll be giving a talk at ApacheCon next month on running
tiered Kafka architectures. If you're in Austin for that, please come by
and check it out.


  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message