kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Yifan Ying <nafan...@gmail.com>
Subject Re: Topic discovery when supporting multiple kafka clusters
Date Tue, 06 Dec 2016 19:05:55 GMT
Thanks Asaf, Aseem.

Assigning topics to only a specific set of brokers will probably cause
uneven traffic and it won't prevent topics to be re-assigned to other
brokers when brokers fail.

Like I said, the original cluster is close to out of resources. I remember
there's some limit on # of partitions that each Kafka cluster can have.
Netflix recommends to keep it below 10k to improve availability and reduce
latency,
http://techblog.netflix.com/2016/04/kafka-inside-keystone-pipeline.html.
Jun Rao also wrote a blog(
https://www.confluent.io/blog/how-to-choose-the-number-of-topicspartitions-in-a-kafka-cluster/)
about how too many partitions could hurt availability and latency. That's
why we want to create another cluster instead of expanding the current one.
I know a lot of companies are maintaining multiple clusters, and I'm
curious how people are doing topic discovery.



On Tue, Dec 6, 2016 at 4:04 AM, Aseem Bansal <asmbansal2@gmail.com> wrote:

> What configurations allow you to assign topics to specific brokers?
>
> I can see https://kafka.apache.org/documentation#basic_ops_automigrate.
> This should allow you to move things around but does that keep anything
> from being re-assigned to the old ones?
>
> On Tue, Dec 6, 2016 at 5:25 PM, Asaf Mesika <asaf.mesika@gmail.com> wrote:
>
> > Why not re-use same cluster? You can assign topics to be live only
> within a
> > specific set of brokers. Thus you have one "bus" for messages,
> simplifying
> > your applications code and configurations
> >
> > On Mon, Dec 5, 2016 at 9:43 PM Yifan Ying <nafan628@gmail.com> wrote:
> >
> > > Hi,
> > >
> > > Initially, we have only one Kafka cluster shared across all teams. But
> > now
> > > this cluster is very close to out of resources (disk space, # of
> > > partitions, etc.). So we are considering adding another Kafka cluster.
> > But
> > > what's the best practice of topic discovery, so that applications know
> > > which cluster their topics live? We have been using Zookeeper for
> service
> > > discovery, maybe it's also good for this purpose?
> > >
> > > Thanks
> > >
> > > --
> > > Yifan
> > >
> >
>



-- 
Yifan

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message