kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Marko Bonaći <marko.bon...@sematext.com>
Subject Re: Maximum Topic Length in Kafka
Date Sun, 29 Nov 2015 15:24:06 GMT
Yes, I thought you weren't interested in retention, but how to limit the
amount of messages produced into a topic.
Take a look at this Kafka Improvement Proposal (KIP):
https://cwiki.apache.org/confluence/display/KAFKA/KIP-13+-+Quotas
But, AFAIK, there's nothing currently available for your use case.

Perhaps you could check Consumer offsets from your Producer and then decide
based on that information whether to throttle Producer or not. Could get
complicated really fast, though.

Marko Bonaći
Monitoring | Alerting | Anomaly Detection | Centralized Log Management
Solr & Elasticsearch Support
Sematext <http://sematext.com/> | Contact
<http://sematext.com/about/contact.html>

On Sun, Nov 29, 2015 at 8:57 AM, Debraj Manna <subharaj.manna@gmail.com>
wrote:

> Let me explain my use case:-
>
> We have a ELK setup in which logstash-forwarders pushes logs from different
> services to a logstash. The logstash then pushes them to kafka. The
> logstash consumer then pulls them out of Kafka and indexes them to
> Elasticsearch cluster.
>
> We are trying to ensure that no single service logs doesn't overwhelm the
> system. So I was thinking if each service logs go in their own topics in
> kafka and if we can specify a maximum length in the topic then the producer
> of that topic can block when a kafka topic is full.
> AFAIK there is no such notion as maximum length of a topic, i.e. offset has
> no limit, except Long.MAX_VALUE I think, which should be enough for a
> couple of lifetimes (9 * 10E18, or quintillion or million trillions).
>
> What would be the purpose of that, besides being a nice foot-gun :)
>
> Marko Bonaći
> Monitoring | Alerting | Anomaly Detection | Centralized Log Management
> Solr & Elasticsearch Support
> Sematext <http://sematext.com/> | Contact
> <http://sematext.com/about/contact.html>
>
> On Sat, Nov 28, 2015 at 2:13 PM, Debraj Manna <subharaj.manna@gmail.com>
> wrote:
>
> > Hi,
> >
> > Can some one please let me know the following:-
> >
> >
> >    1. Is it possible to specify maximum length of a particular topic ( in
> >    terms of number of messages ) in kafka ?
> >    2. Also how does Kafka behave when a particular topic gets full?
> >    3. Can the producer be blocked if a topic get full rather than
> deleting
> >    old messages?
> >
> > I have gone through the documentation
> > <http://kafka.apache.org/081/documentation.html#basic_ops_add_topic> but
> > could not find anything of what I am looking for.
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message