kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jay Kreps <jay.kr...@gmail.com>
Subject Re: Strategies for high-concurrency consumers
Date Thu, 06 Nov 2014 19:31:02 GMT
Unfortunately the performance of the consumer balancing scales poorly with
the number of partitions. This is one of the things the consumer rewrite
project is meant to address, however that is not complete yet. A reasonable
workaround may be to decouple your application parallelism from the number
of partitions. I.e. have the processing of each partition happen in a
threadpool. I'm assuming that you don't actually have 2,500 machines, just
that you need that much parallelism since each messages takes a bit of time
to process. This does weaken the delivery ordering, but you may be able to
shard the processing by key to solve that problem.


On Thu, Nov 6, 2014 at 10:59 AM, Jack Foy <jfoy@whitepages.com> wrote:

> Hi all,
> We are building a system that will carry a high volume of traffic (on the
> order of 2 billion messages in each batch), which we need to process at a
> rate of 50,000 messages per second. We need to guarantee at-least-once
> delivery for each message. The system we are feeding has a latency of 50ms
> per message, and can absorb many concurrent requests.
> We have a Kafka cluster with three brokers and a Zookeeper 3.4.5
> cluster with 5 nodes, each on physical hardware.
> We intend to deploy a consumer group of 2500 consumers against a single
> topic, with a partition for each consumer. We expect our consumers to be
> stable over the course of the run, so we expect rebalancing to be rare. In
> testing, we have successfully run 512 high-level consumers against 1024
> partitions, but beyond 512 consumers the rebalance at startup doesn’t
> complete within 10 minutes. Is this a workable strategy with high-level
> consumers? Can we actually deploy a consumer group with this many consumers
> and partitions?
> We see throughput of more than 500,000 messages per second with our 512
> consumers, but we need greater parallelism to meet our performance needs.
> --
> Jack Foy <jfoy@whitepages.com>

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message