kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Benjamin Manns <benma...@gmail.com>
Subject Re: max number of partitions with v0.9.0
Date Wed, 04 May 2016 03:42:18 GMT
>From my knowledge (beginner's) each partition still requires at least a
file selector on the Kafka brokers. The new consumer structure means
consumers won't store data in Zookeeper, but topics and partitions still do.

What I would do is key by your ID and place a rate limiting stream
processor in front of your heavier processors. This could be a windowed
task that counts how many messages have been sent in the last few seconds
or minutes. For under-limit IDs send to a high priority topic. For over
limit, a lower priority topic.


On Tuesday, May 3, 2016, David Shepherd <dtshepherd@gmail.com> wrote:

> I was wonder if the new Kafka Consumer introduced in 0.9.0 (
> http://www.confluent.io/blog/tutorial-getting-started-with-the-new-apache-kafka-0.9-consumer-client
> )
> allows for a higher number of partitions in a given cluster since it
> removes the zookeeper dependency. I understand the file descriptor and
> availability concerns discussed here:
> http://www.confluent.io/blog/how-to-choose-the-number-of-topicspartitions-in-a-kafka-cluster/
> .
> The reason I ask is because we'd like to use partitioning to limit the
> impact of a message flood on our downstream consumers. If we can partition
> by a particular ID, it will isolate message floods from a given source into
> a single partition, which allows us to allocate a single consume to process
> that flood without affecting quality of service to the rest of the system.
> Unfortunately, partitioning this way could create millions of partitions,
> each only producing a few messages per minute with the exception that a few
> of the partitions will be sending thousands of messages per minute.
> I'm also open to suggestions on how others have solved the flooding / noisy
> neighbor problem in Kafka.
> Thanks,
> Dave Shepherd

Benjamin Manns
(434) 321-8324

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message