kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Wes Chow <...@chartbeat.com>
Subject Re: HDD or SSD or EBS for kafka brokers in Amazon EC2
Date Tue, 02 Jun 2015 19:31:58 GMT

Our workaround is to switch to i2's. Amazon didn't mention anything, 
though we're getting on a call with them soon so I'll be sure to ask. 
Fwiw, we're also on 12.04.

Wes


> Daniel Nelson <mailto:daniel.nelson@vungle.com>
> June 2, 2015 at 2:42 PM
>
> Do you have any workarounds for the d2 issues? We’ve been using them 
> for our Kafkas too, and ran into the instability. We’re on Ubuntu 
> 12.04 and plan to try on 14.04 with the latest HWE to see if that 
> helps any.
>
> Thanks!
> Wes Chow <mailto:wes@chartbeat.com>
> June 2, 2015 at 1:39 PM
>
> We have run d2 instances with Kafka. They're currently unstable -- 
> Amazon confirmed a host issue with d2 instances that gets tickled by a 
> Kafka workload yesterday. Otherwise, it seems the d2 instance type is 
> ideal as it gets an enormous amount of disk throughput and you'll 
> likely be network bottlenecked.
>
> Wes
>
>
> Henry Cai <mailto:hcai@pinterest.com.INVALID>
> June 2, 2015 at 12:37 PM
> We have been hosting kafka brokers in Amazon EC2 and we are using EBS
> disk. But periodically we were hit by long I/O wait time on EBS in some
> Availability Zones.
>
> We are thinking to change the instance types to a local HDD or local SSD.
> HDD is cheaper and bigger and seems quite fit for the Kafka use case which
> is mostly sequential read/write, but some early experiments show the HDD
> cannot catch up with the message producing speed since there are many
> topic/partitions on the broker which actually makes the disk I/O more
> randomly accessed.
>
> How are people's experience of choosing disk types on Amazon?
>

Mime
View raw message