kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From mission mission <mission0...@gmail.com>
Subject Trade-off between topics and partitions?
Date Fri, 06 Dec 2013 03:47:19 GMT
Hello,

According to the Kafka FAQ "How do I choose the number of partitions for a
topic", clusters with more than 10K partitions are not tested. I am looking
for advice on how to scale the number of partitions beyond that. My use
case is to publish messages to 1 million users, each with an unique user
id. Users are not always connected but a user must receive published
messages in order.

What is the best way to divide topics and partitions for this use case? Do
I need 1 million partitions? The FAQ seems to think so, i.e. "if we were
storing notifications for users we would encourage a design with a single
notifications topic partitioned by user id". But the FAQ implies strongly
that 1 million partitions may wreak havoc on zookeeper because they will
lead to X million znodes that have to be stored in memory. Any suggestions?

Thanks,

mission

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message