kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Emmanuel <ele...@msn.com>
Subject RE: Post on running Kafka at LinkedIn
Date Fri, 20 Mar 2015 22:03:29 GMT
800B messages / day = 9.26M messages / sec over 1100 brokers 
= ~8400 message / broker / sec
Do I get this right?
Trying to benchmark my own test cluster and that's what I see with 2 brokers...Just wondering
if my numbers are good or bad...


> Subject: Re: Post on running Kafka at LinkedIn
> From: clark@kafka.guru
> Date: Fri, 20 Mar 2015 14:27:58 -0700
> To: users@kafka.apache.org
> 
> Yep! We are growing :)
> 
> -Clark
> 
> Sent from my iPhone
> 
> > On Mar 20, 2015, at 2:14 PM, James Cheng <jcheng@tivo.com> wrote:
> > 
> > Amazing growth numbers.
> > 
> > At the meetup on 1/27, Clark Haskins presented their Kafka usage at the time. It
was:
> > 
> > Bytes in: 120 TB
> > Messages In: 585 million
> > Bytes out: 540 TB
> > Total brokers: 704
> > 
> > In Todd's post, the current numbers:
> > 
> > Bytes in: 175 TB (45% growth)
> > Messages In: 800 billion (36% growth)
> > Bytes out: 650 TB (20% growth)
> > Total brokers: 1100 (56% growth)
> > 
> > That much growth in just 2 months? Wowzers.
> > 
> > -James
> > 
> >> On Mar 20, 2015, at 11:30 AM, James Cheng <jcheng@tivo.com> wrote:
> >> 
> >> For those who missed it:
> >> 
> >> The Kafka Audit tool was also presented at the 1/27 Kafka meetup:
> >> http://www.meetup.com/http-kafka-apache-org/events/219626780/
> >> 
> >> Recorded video is here, starting around the 40 minute mark:
> >> http://www.ustream.tv/recorded/58109076
> >> 
> >> Slides are here:
> >> http://www.ustream.tv/recorded/58109076
> >> 
> >> -James
> >> 
> >>> On Mar 20, 2015, at 9:47 AM, Todd Palino <tpalino@gmail.com> wrote:
> >>> 
> >>> For those who are interested in detail on how we've got Kafka set up at
> >>> LinkedIn, I have just published a new posted to our Engineering blog titled
> >>> "Running Kafka at Scale"
> >>> 
> >>>   https://engineering.linkedin.com/kafka/running-kafka-scale
> >>> 
> >>> It's a general overview of our current Kafka install, tiered architecture,
> >>> audit, and the libraries we use for producers and consumers. You'll also
be
> >>> seeing more posts from the SRE team here in the coming weeks on deeper
> >>> looks into both Kafka and Samza.
> >>> 
> >>> Additionally, I'll be giving a talk at ApacheCon next month on running
> >>> tiered Kafka architectures. If you're in Austin for that, please come by
> >>> and check it out.
> >>> 
> >>> -Todd
> > 
 		 	   		  
Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message