kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chen Wang <chen.apache.s...@gmail.com>
Subject Re: Issue with 240 topics per day
Date Mon, 11 Aug 2014 23:45:22 GMT
Todd,
I actually only intend to keep each topic valid for 3 days most. Each of
our topic has 3 partitions, so its around 3*240*3 =2160 partitions. Since
there is no api for deleting topic, i guess i could set up a cron job
deleting the out dated topics(folders) from zookeeper..
do you know when the delete topic api will be available in kafka?
Chen


On Mon, Aug 11, 2014 at 3:47 PM, Todd Palino <tpalino@linkedin.com.invalid>
wrote:

> You need to consider your total partition count as you do this. After 30
> days, assuming 1 partition per topic, you have 7200 partitions. Depending
> on how many brokers you have, this can start to be a problem. We just
> found an issue on one of our clusters that has over 70k partitions that
> there┬╣s now a problem with doing actions like a preferred replica election
> for all topics because the JSON object that gets written to the zookeeper
> node to trigger it is too large for Zookeeper┬╣s default 1 MB data size.
>
> You also need to think about the number of open file handles. Even with no
> data, there will be open files for each topic.
>
> -Todd
>
>
> On 8/11/14, 2:19 PM, "Chen Wang" <chen.apache.solr@gmail.com> wrote:
>
> >Folks,
> >Is there any potential issue with creating 240 topics every day? Although
> >the retention of each topic is set to be 2 days, I am a little concerned
> >that since right now there is no delete topic api, the zookeepers might be
> >overloaded.
> >Thanks,
> >Chen
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message