kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Dave Mangot <dman...@librato.com>
Subject Kafka on EBS st1?
Date Tue, 04 Oct 2016 18:01:49 GMT
Has anyone been running kafka well on the st1 EBS volumes?

We've historically run on m1 and m2 instance types for our Kafka workload
but wanted to move to the M4s to get better price/performance.

We rolled out a single instance in two environments with M4 and 1 TB of
st1.  Everything seemed to be going well, lower cpu util, flush times
looked good, etc.   Then, a few hours later, both our Kafka flush times and
CPU wait times went up much higher than the rest of the cluster and just
stayed there.

Looking at the cloudwatch metrics, it shows that our "burst balance" had
been slowly degrading over those hours, and as soon as it was exhausted,
that's when the elevated times happened.

I could possibly believe that with the production workload, we were
overwhelming some allocation, but in our staging environment, it makes no
sense.   Cloudwatch says that our average write size is 40 KB/op which I
suspect is just far too low for the st1 as it's designed for large
sequential writes.  I believe we're eating up our IOPS allocation but I
could be just looking at the completely wrong thing.

We're using an XFS filesystem with a 16 M allocsize.

Does anyone have experience with this volume type?  I suspect we're just
holding it wrong.

Cheers,

-Dave


-- 
Dave Mangot
Director of Operations
Librato and Papertrail

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message