kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Marko Bonaći <marko.bon...@sematext.com>
Subject Re: Maximum Topic Length in Kafka
Date Sun, 29 Nov 2015 17:19:29 GMT
Did not know that quotas landed in 0.9. Very nice!
Being able to throttle clients that don't have real-time SLAs (in favor of
those who do) is a great addition.
Thanks for that Grant.

Marko Bonaći
Monitoring | Alerting | Anomaly Detection | Centralized Log Management
Solr & Elasticsearch Support
Sematext <http://sematext.com/> | Contact
<http://sematext.com/about/contact.html>

On Sun, Nov 29, 2015 at 5:05 PM, Grant Henke <ghenke@cloudera.com> wrote:

> Quotas (KIP-13) is actually included in the recent 0.9.0 release. More
> about them can be read in the documentation here:
>
>    - http://kafka.apache.org/documentation.html#design_quotas
>    - http://kafka.apache.org/documentation.html#quotas
>
>
>
> On Sun, Nov 29, 2015 at 9:24 AM, Marko Bonaći <marko.bonaci@sematext.com>
> wrote:
>
> > Yes, I thought you weren't interested in retention, but how to limit the
> > amount of messages produced into a topic.
> > Take a look at this Kafka Improvement Proposal (KIP):
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-13+-+Quotas
> > But, AFAIK, there's nothing currently available for your use case.
> >
> > Perhaps you could check Consumer offsets from your Producer and then
> decide
> > based on that information whether to throttle Producer or not. Could get
> > complicated really fast, though.
> >
> > Marko Bonaći
> > Monitoring | Alerting | Anomaly Detection | Centralized Log Management
> > Solr & Elasticsearch Support
> > Sematext <http://sematext.com/> | Contact
> > <http://sematext.com/about/contact.html>
> >
> > On Sun, Nov 29, 2015 at 8:57 AM, Debraj Manna <subharaj.manna@gmail.com>
> > wrote:
> >
> > > Let me explain my use case:-
> > >
> > > We have a ELK setup in which logstash-forwarders pushes logs from
> > different
> > > services to a logstash. The logstash then pushes them to kafka. The
> > > logstash consumer then pulls them out of Kafka and indexes them to
> > > Elasticsearch cluster.
> > >
> > > We are trying to ensure that no single service logs doesn't overwhelm
> the
> > > system. So I was thinking if each service logs go in their own topics
> in
> > > kafka and if we can specify a maximum length in the topic then the
> > producer
> > > of that topic can block when a kafka topic is full.
> > > AFAIK there is no such notion as maximum length of a topic, i.e. offset
> > has
> > > no limit, except Long.MAX_VALUE I think, which should be enough for a
> > > couple of lifetimes (9 * 10E18, or quintillion or million trillions).
> > >
> > > What would be the purpose of that, besides being a nice foot-gun :)
> > >
> > > Marko Bonaći
> > > Monitoring | Alerting | Anomaly Detection | Centralized Log Management
> > > Solr & Elasticsearch Support
> > > Sematext <http://sematext.com/> | Contact
> > > <http://sematext.com/about/contact.html>
> > >
> > > On Sat, Nov 28, 2015 at 2:13 PM, Debraj Manna <
> subharaj.manna@gmail.com>
> > > wrote:
> > >
> > > > Hi,
> > > >
> > > > Can some one please let me know the following:-
> > > >
> > > >
> > > >    1. Is it possible to specify maximum length of a particular topic
> (
> > in
> > > >    terms of number of messages ) in kafka ?
> > > >    2. Also how does Kafka behave when a particular topic gets full?
> > > >    3. Can the producer be blocked if a topic get full rather than
> > > deleting
> > > >    old messages?
> > > >
> > > > I have gone through the documentation
> > > > <http://kafka.apache.org/081/documentation.html#basic_ops_add_topic>
> > but
> > > > could not find anything of what I am looking for.
> > > >
> > >
> >
>
>
>
> --
> Grant Henke
> Software Engineer | Cloudera
> grant@cloudera.com | twitter.com/gchenke | linkedin.com/in/granthenke
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message