kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jay Kreps <jay.kr...@gmail.com>
Subject Re: priority of topic
Date Tue, 12 Jun 2012 15:15:23 GMT
This does make sense, but I wonder if it would be very usable or worth the
effort? Maybe the solution is usually to just have good monitoring and have
sufficient capacity for your workload--that is rather than trying to handle
overload more gracefully, just avoid it by adding servers.

In any case, here would be one implementation approach:
Currently (as of 0.8) we keep a queue of requests waiting for processing by
one of N I/O threads (RequestChannel.scala). Currently this is just a
linear blocking queue. We could make it into a priority queue and assign a
priority to each request (either based on the topic or on a per-request
basis). Instead of processing requests on a FIFO basis we would always
process the highest priority request. The problem with this, of course, is
that it introduces all the scheduling problems common to OS resource
schedulers. For example if there is a continuous stream of high priority
requests it will starve out the lower priority request from EVER getting
processed, which may or may not be desired. This also complicates the
ordering guarantees. Since now two messages for the same topic/partition
would only be appended in-order if they are of exactly the same priority.


On Mon, Jun 11, 2012 at 5:08 PM, xiaoyu wang <xiaoyu.wang@gmail.com> wrote:

> Hi all,
> Just want to bring up some discussion, does it make sense to have priority
> defined over topics? In case of resource contention, topic with higher
> priority get delivered first.
> -Xiaoyu

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message