kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From James Cheng <jch...@tivo.com>
Subject Re: Post on running Kafka at LinkedIn
Date Fri, 20 Mar 2015 21:14:18 GMT
Amazing growth numbers.

At the meetup on 1/27, Clark Haskins presented their Kafka usage at the time. It was:

Bytes in: 120 TB
Messages In: 585 million
Bytes out: 540 TB
Total brokers: 704

In Todd's post, the current numbers:

Bytes in: 175 TB (45% growth)
Messages In: 800 billion (36% growth)
Bytes out: 650 TB (20% growth)
Total brokers: 1100 (56% growth)

That much growth in just 2 months? Wowzers.

-James

> On Mar 20, 2015, at 11:30 AM, James Cheng <jcheng@tivo.com> wrote:
> 
> For those who missed it:
> 
> The Kafka Audit tool was also presented at the 1/27 Kafka meetup:
> http://www.meetup.com/http-kafka-apache-org/events/219626780/
> 
> Recorded video is here, starting around the 40 minute mark:
> http://www.ustream.tv/recorded/58109076
> 
> Slides are here:
> http://www.ustream.tv/recorded/58109076
> 
> -James
> 
>> On Mar 20, 2015, at 9:47 AM, Todd Palino <tpalino@gmail.com> wrote:
>> 
>> For those who are interested in detail on how we've got Kafka set up at
>> LinkedIn, I have just published a new posted to our Engineering blog titled
>> "Running Kafka at Scale"
>> 
>>    https://engineering.linkedin.com/kafka/running-kafka-scale
>> 
>> It's a general overview of our current Kafka install, tiered architecture,
>> audit, and the libraries we use for producers and consumers. You'll also be
>> seeing more posts from the SRE team here in the coming weeks on deeper
>> looks into both Kafka and Samza.
>> 
>> Additionally, I'll be giving a talk at ApacheCon next month on running
>> tiered Kafka architectures. If you're in Austin for that, please come by
>> and check it out.
>> 
>> -Todd
> 


Mime
View raw message