kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Suman B N <sumannew...@gmail.com>
Subject Re: High write operations rate on disk
Date Tue, 07 Apr 2020 07:28:03 GMT
We have used st1 volumes and we never saw any issue.
Yes, we are using m-series. Even t-series worked for us :D

During those spikes, do you observe any background operations going on?
Check server logs, controller logs.

On Tue, Apr 7, 2020 at 12:49 PM Seva Feldman <seva.f@ironsrc.com> wrote:

> ST1 EBS fit only for sequential rights and reads. Once you have many
> partitions on EBS it will be mostly random.
> Interesting to monitor random vs sequential...
>
> We tested kafka on ST1 with 1xx partitions on each EBS and it was
> constantly lagging.
>
> BR
>
> On Tue, Apr 7, 2020 at 10:06 AM Soumyajit Sahu <soumyajit.sahu@gmail.com>
> wrote:
>
> > Our typical IOPS stays at ~10K write ops/min, but it goes to 37K write
> > ops/min (which is where AWS throttles).
> > The spike in write ops isn't accompanied by any spike in write throughput
> > or produce requests (except for the first few minutes of catch up). The
> > write ops spike stays up (persistently for an hour or two) until we stop
> > the broker ec2 instance for about 30 mins and then start it back.
> >
> > @Liam, no, we are not using log compaction except for a few consumer
> offset
> > topics and config topic (for Kafka Connect), and schema registry store.
> >
> > @Suman, are you using m5 or r5 instances. Recently, we migrated from r5
> to
> > m5, and I wonder if that has a hand in this.
> >
> > We have about 1000 partitions residing on each disk, but I don't think
> that
> > matters as most of the time the brokers run flawlessly (even during peak
> > traffic hours).
> >
> > Thanks!
> >
> > On Mon, Apr 6, 2020 at 11:39 PM Suman B N <sumannewton@gmail.com> wrote:
> >
> > > We too have a similar setup but we never observed any such spikes.
> > >
> > > Are you sure your disk IOPS is good enough? Check if that is
> throttling.
> > >
> > > After a broker restarts, there might be more traffic as well because of
> > > followers trying to catch up with the leader.
> > >
> > > -Suman
> > >
> > > On Tue, Apr 7, 2020 at 11:59 AM Soumyajit Sahu <
> soumyajit.sahu@gmail.com
> > >
> > > wrote:
> > >
> > > > We are running Kafka on AWS EC2 instances (m5.2xlarge) with mounted
> EBS
> > > st1
> > > > volume (one on each machine).
> > > > Occasionally, we have noticed that the write ops/second goes through
> > the
> > > > roof and we get throttled by AWS while the data throughput wouldn't
> > have
> > > > changed much. As far as our observation goes, it happens usually
> after
> > a
> > > > broker restart.
> > > >
> > > > Has anyone else come across this behavior?
> > > >
> > > > Thanks!
> > > >
> > >
> > >
> > > --
> > > *Suman*
> > > *OlaCabs*
> > >
> >
>
>
> --
> Seva Feldman
> VP R&D Mobile Delivery
> [image: ironSource] <http://www.ironsrc.com/>
>
> email seva.f@ironsrc.com
> mobile +972544346089
>
> ironSource HQ - 121 Derech Menachem Begin st. Tel Aviv
>


-- 
*Suman*
*OlaCabs*

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message