kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chen Wang <chen.apache.s...@gmail.com>
Subject Re: Issue with 240 topics per day
Date Tue, 12 Aug 2014 00:18:56 GMT
Philip,
That is right. There is huge amount of data flushed into the topic within
each 6 minutes. Then at the end of each 6 min, I only want to read from
that specify topic, and data within that topic has to be processed as fast
as possible. I was originally using redis queue for this purpose, but it
takes much longer to process a redis queue than kafka queue(testing data is
2M messages). Since we already have kafka infrastructure setup, instead of
seeking other tools(activeMQ, rabbitMQ etc), I would rather make use of
kafka, although it does not seem like a common kafka user case.

Chen


On Mon, Aug 11, 2014 at 5:01 PM, Philip O'Toole <
philip.otoole@yahoo.com.invalid> wrote:

> I'd love to know more about what you're trying to do here. It sounds like
> you're trying to create topics on a schedule, trying to make it easy to
> locate data for a given time range? I'm not sure it makes sense to use
> Kafka in this manner.
>
> Can you provide more detail?
>
>
> Philip
>
>
> -----------------------------------------
> http://www.philipotoole.com
>
>
> On Monday, August 11, 2014 4:45 PM, Chen Wang <chen.apache.solr@gmail.com>
> wrote:
>
>
>
> Todd,
> I actually only intend to keep each topic valid for 3 days most. Each of
> our topic has 3 partitions, so its around 3*240*3 =2160 partitions. Since
> there is no api for deleting topic, i guess i could set up a cron job
> deleting the out dated topics(folders) from zookeeper..
> do you know when the delete topic api will be available in kafka?
> Chen
>
>
>
> On Mon, Aug 11, 2014 at 3:47 PM, Todd Palino <tpalino@linkedin.com.invalid
> >
> wrote:
>
> > You need to consider your total partition count as you do this. After 30
> > days, assuming 1 partition per topic, you have 7200 partitions. Depending
> > on how many brokers you have, this can start to be a problem. We just
> > found an issue on one of our clusters that has over 70k partitions that
> > there┬╣s now a problem with doing actions like a preferred replica
> election
> > for all topics because the JSON object that gets written to the zookeeper
> > node to trigger it is too large for Zookeeper┬╣s default 1 MB data size.
> >
> > You also need to think about the number of open file handles. Even with
> no
> > data, there will be open files for each topic.
> >
> > -Todd
> >
> >
> > On 8/11/14, 2:19 PM, "Chen Wang" <chen.apache.solr@gmail.com> wrote:
> >
> > >Folks,
> > >Is there any potential issue with creating 240 topics every day?
> Although
> > >the retention of each topic is set to be 2 days, I am a little concerned
> > >that since right now there is no delete topic api, the zookeepers might
> be
> > >overloaded.
> > >Thanks,
> > >Chen
> >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message