kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jason Gustafson <ja...@confluent.io>
Subject Re: Consumer limit for pub-sub mode
Date Mon, 03 Aug 2015 17:20:29 GMT
Hey Valibhav,

With only one partition, all of the consumers will end up hitting a single
broker (since partitions cannot be split). Whether it is possible to get
that number of consumers on a single broker may depend on the message load
through the topic. I think there has been some interest in allowing
consumers to read from partition replicas and that would help, but I'm not
sure if there is anything solid. If the single broker is not enough to
handle the load, then you might also consider replicating the topic to
several other topics in the same cluster (e.g. using mirror maker), and
that would allow you to distribute the consumer load across those other
topics. Others might have better ideas.

-Jason

On Sun, Aug 2, 2015 at 10:09 PM, Vaibhav Kirte <vaibhav.kirte@sokrati.com>
wrote:

> Hi,
>
> I need to know how many consumers can subscribe to a single topic ( with
> one partition ).
>
> I have a requirement such that,
>     1. The producer will post to 1 topic having 1 partition.
>     1. *20,000-30,000 consumers *should to be able to consume messages.
>     2. All of the consumers should receive all messages that are produced.
>
> will this be possible using kafka ?
> is there a limit on number of consumers ?
> what will be the number of machines that I will need to satisfy the
> requirements ?
>
> --
> Regards,
> Vaibhav Kirte
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message