kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jason Rosenberg <...@squareup.com>
Subject Re: implications of using large number of topics....
Date Sun, 14 Oct 2012 06:42:59 GMT
Cool,

What's the schedule for 0.8 coming out?  Are there any pre-release versions?

Jason

On Sat, Oct 13, 2012 at 8:37 PM, Jun Rao <junrao@gmail.com> wrote:

> Jason,
>
> The issue with 0.7 is that a topic exists on every broker and every time
> one adds a new broker, some additional partitions for each existing topic
> are added to the new broker. This is going to change in 0.8. A topic has a
> fixed number of partitions, independent of the # of brokers. So, by adding
> more brokers, we can support more topics in a cluster.
>
> Thanks,
>
> Jun
>
> On Fri, Oct 12, 2012 at 10:55 AM, Jason Rosenberg <jbr@squareup.com>
> wrote:
>
> > Has there ever been a thought to better handle a large number of topics?
> >  Prior discussions?  Or would it likely be too great of a change to the
> way
> > kafka works, no matter what?
> >
> > I'm wondering if there's a way to have a notion of multiple "virtual"
> > topics which are internally managed as members of a single topic "group",
> > but which at the api level, appear to be unique topics, from the client
> > perspective.
> >
> > Naturally, it would be straightforward to implement something like this
> by
> > wrapping the current client apis, but I'm wondering if there's any
> benefit
> > to building it into the internals.  This would still have the downside
> that
> > a client subscribing to a virtual topic would have to, under the covers,
> > sift through lots of messages it's not interested in.
> >
> > Any other interesting approaches?
> >
> > Jason
> >
> >
> > On Thu, Oct 11, 2012 at 10:48 PM, Jun Rao <junrao@gmail.com> wrote:
> >
> > > Mathias,
> > >
> > > What matters is the total # partitions since each corresponds to a
> > separate
> > > directory on disk. It doesn't matter how may topics those partitions
> are
> > > from.
> > >
> > > Thanks,
> > >
> > > Jun
> > >
> > > On Thu, Oct 11, 2012 at 6:43 AM, Mathias Söderberg <
> > > mathias.soederberg@gmail.com> wrote:
> > >
> > > > Hey all,
> > > >
> > > > This is a quite interesting topic (no pun intended), and I've seen it
> > > come
> > > > up at least once before.
> > > >
> > > > Me and a friend started experimenting with Kafka and ZooKeeper a
> little
> > > > while ago (building a publisher / subscriber system with consistent
> > > hashing
> > > > and whatnot) and currently we're using around 300 topics, all with
> one
> > > > partition each. So far we haven't really done any serious performance
> > > > testing, but I'm planning to do so in the following weeks. But I've
> > got a
> > > > few questions regardless:
> > > >
> > > >
> > > > Does / should it make any difference in performance when one has a
> lot
> > of
> > > > topics compared to having one topic with a lot of partitions? I'm
> > > imagining
> > > > that the system still needs to keep the same number of file
> descriptors
> > > > open, but I'm not sure how this would affect reads and writes? Are we
> > > going
> > > > to run into more random reads and writes by using a lot of topics
> > > compared
> > > > to using one topic with a lot of partitions instead? Can't really
> wrap
> > my
> > > > head around this right now, mostly because of my rather limited
> > knowledge
> > > > about how disks and page caches work.
> > > >
> > > > Could also add that we're mostly doing sequential reads (in rare
> cases
> > we
> > > > have to rewind a topic) and that the number of topics doesn't change.
> > > >
> > > > On 11 October 2012 05:13, Taylor Gautier <tgautier@gmail.com> wrote:
> > > >
> > > > > We used pattern #1 at Tagged.  I wouldn't recommend it unless
> you're
> > > > really
> > > > > committed.  It took a lot of work to get it working right.
> > > > >
> > > > > a) Performance degraded non-linearly (read it fell off a cliff)
> when
> > > > > brokers were managing more than about 20k topics.  This was on a
> > Linux
> > > > RHEL
> > > > > 5.3 system with EXT3.  YMMV.
> > > > >
> > > > > b) Startup time is significantly longer for a broker that is
> > restarted
> > > > due
> > > > > to communication with ZK to sync up on those topics.
> > > > >
> > > > > c) If topics are short lived, even if Kafka expires the data
> segments
> > > > using
> > > > > it's standard 0.7 cleaner, the directory name for the topic will
> > still
> > > > > exist on disk and the topic is still considered "active" (in
> memory)
> > in
> > > > > Kafka.  This causes problems - see a above (open file handles).
> > > > >
> > > > > d) Message latency is affected.  Kafka syncs messages to disk if
x
> > > > messages
> > > > > have buffered in memory, or y seconds have elapsed (both
> > configurable).
> > > >  If
> > > > > you have few topics and many messages (pattern #2), you will be
> > hitting
> > > > the
> > > > > x limit quite often, and get good throughput.  However, if you have
> > > many
> > > > > topics and few messages per topic (pattern #1), you will have to
> rely
> > > on
> > > > > the y threshold to flush to disk, and setting this too low can
> impact
> > > > > performance (throughput) in a significant way.  Jay already
> mentioned
> > > > this
> > > > > as random writes.
> > > > >
> > > > > We had to implement a number of solutions ourselves to resolve
> these
> > > > > issues, namely:
> > > > >
> > > > > #1 No ZK.  This means that all of the automatic partitioning done
> by
> > > > Kafka
> > > > > is not available, so we had to roll our own (luckily Tagged is
> pretty
> > > > used
> > > > > to scaling systems so there was much in-house expertise).  The
> > solution
> > > > > here was to implement a R/W proxy layer of machines to intercept
> > > messages
> > > > > and read/write to/from Kafka handling the sharding at the proxy
> > layer.
> > > > >  Because most of our messages were coming from PHP and we didn't
> want
> > > to
> > > > > use TCP we needed a UDP/TCP bridge/proxy anyway so this wasn't a
> huge
> > > > deal
> > > > > (also, we wanted strict ordering of messages, so we needed a shard
> by
> > > > topic
> > > > > feature anyway (I believe this can be done in 0.7 but we started
> with
> > > > 0.6)
> > > > >
> > > > > #2 Custom cleaner.  We implemented an extra cleanup task inside the
> > > Kafka
> > > > > process that could completely remove a topic from memory and disk.
> >  For
> > > > > clients, this sometimes meant that a subscribed topic suddenly
> > changed
> > > > it's
> > > > > physical offset from some offset X to 0, but that's ok, while
> > > technically
> > > > > it probably would never happen theoretically clients should have
to
> > > > handle
> > > > > this case anyway because the Kafka physical message space is
> limited
> > to
> > > > > 64-bits (again, unlikely to ever wrap in practice, but you never
> > know).
> > > > >  Anyway it's pretty easy to handle this just catch the "invalid
> > offset"
> > > > > error Kafka gives and start at 0.
> > > > >
> > > > > #3 Low threshold for flush.  This gave us good latency, but poor
> > > > throughput
> > > > > (relatively speaking).  We had more than enough throughput, but it
> > was
> > > > > nowhere near what Kafka can do when setup in pattern #1.
> > > > >
> > > > > Given that you want to manage "hundreds of thousands of topics"
> that
> > > may
> > > > > mean that you would need 10's of Kafka brokers which could be
> another
> > > > > source of problems - it's more cost, more management, and more
> > sources
> > > of
> > > > > failure.  SSD's may help solve this problem btw, but now you are
> > > talking
> > > > > expensive machines rather than using just off the shelf cheapo
> > servers
> > > > with
> > > > > standard SATA drives.
> > > > >
> > > > > On Wed, Oct 10, 2012 at 4:25 PM, Jay Kreps <jay.kreps@gmail.com>
> > > wrote:
> > > > >
> > > > > > Yes the footprint of a topic is one directory per partition
(a
> > topic
> > > > can
> > > > > > have many subpartitions per partitions). Each directory contains
> > one
> > > or
> > > > > > more files (depending on how much data you are retaining and
the
> > > > segment
> > > > > > size, both configurable).
> > > > > >
> > > > > > In addition to having lots of open files, which certainly scales
> up
> > > to
> > > > > the
> > > > > > hundreds of thousands, this will also impact the I/O pattern.
As
> > the
> > > > > number
> > > > > > of files increases the data written to each file necessarily
> > > decreases.
> > > > > > This likely means lots of random I/O. The OS can group together
> > > writes,
> > > > > but
> > > > > > if you only doing a single write per topic every now and then
> there
> > > > will
> > > > > be
> > > > > > nothing to group and you will lots of small random I/O. This
will
> > > > > > definitely impact throughput. I don't know where the practical
> > limits
> > > > are
> > > > > > we have tested up to ~500 topics and see reasonable performance.
> We
> > > > have
> > > > > > not done serious performance testing with tens of thousands
of
> > topics
> > > > or
> > > > > > more.
> > > > > >
> > > > > > In addition to the filesystem concerns there is metadata kept
for
> > > each
> > > > > > partition in zk, and I believe zk keeps this metadata in memory.
> > > > > >
> > > > > > -Jay
> > > > > >
> > > > > > On Wed, Oct 10, 2012 at 4:12 PM, Jason Rosenberg <
> jbr@squareup.com
> > >
> > > > > wrote:
> > > > > >
> > > > > > > Ok,
> > > > > > >
> > > > > > > Perhaps for the sake of argument, consider the question
if we
> > have
> > > > > just 1
> > > > > > > kafka broker.  It sounds like it will need to keep a file
> handle
> > > open
> > > > > for
> > > > > > > each topic?  Is that right?
> > > > > > >
> > > > > > > Jason
> > > > > > >
> > > > > > > On Wed, Oct 10, 2012 at 4:05 PM, Neha Narkhede <
> > > > > neha.narkhede@gmail.com
> > > > > > > >wrote:
> > > > > > >
> > > > > > > > Hi Jason,
> > > > > > > >
> > > > > > > > We use option #2 at LinkedIn for metrics and tracking
data.
> > > > > Supporting
> > > > > > > > Option #1 in Kafka 0.7 has its challenges since every
topic
> is
> > > > stored
> > > > > > > > on every broker, by design. Hence, the number of topics
a
> > cluster
> > > > can
> > > > > > > > support is limited by the IO and number of open file
handles
> on
> > > > each
> > > > > > > > broker. After Kafka 0.8 is released, the distribution
of
> topics
> > > to
> > > > > > > > brokers is user defined and can scale out with the
number of
> > > > brokers.
> > > > > > > > Having said that, some Kafka users have successfully
deployed
> > > Kafka
> > > > > > > > 0.7 clusters hosting very high number of topics. I
hope they
> > can
> > > > > share
> > > > > > > > their experiences here.
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > > Neha
> > > > > > > >
> > > > > > > > On Wed, Oct 10, 2012 at 3:57 PM, Jason Rosenberg <
> > > jbr@squareup.com
> > > > >
> > > > > > > wrote:
> > > > > > > > > Hi,
> > > > > > > > >
> > > > > > > > > I'm exploring using kafka for the first time.
> > > > > > > > >
> > > > > > > > > I'm contemplating a system where we transmit
metric data at
> > > > regular
> > > > > > > > > intervals to kafka.  One question I have is whether
to
> > generate
> > > > > > simple
> > > > > > > > > messages with very little meta data (just timestamp
and
> > value),
> > > > and
> > > > > > > > keeping
> > > > > > > > > meta data like the name/host/app that generated
metric out
> of
> > > the
> > > > > > > > message,
> > > > > > > > > and have that be embodied in the name of the
topic itself
> > > > instead.
> > > > > > > > >  Alternatively, we could have a relatively small
number of
> > > > topics,
> > > > > > > which
> > > > > > > > > contain messages which include source meta data
along with
> > the
> > > > > > > timestamp
> > > > > > > > > and metric value in each message.
> > > > > > > > >
> > > > > > > > > 1. On one hand, we'd have a large number of topics
(say
> > several
> > > > > > hundred
> > > > > > > > > thousand topics) with small messages, generated
at a steady
> > > rate
> > > > > (say
> > > > > > > one
> > > > > > > > > every 10 seconds).
> > > > > > > > >
> > > > > > > > > 2. Alternatively, we could have just few topics,
which
> > receive
> > > > > > several
> > > > > > > > > hundred thousand messages every 10 seconds, which
contain 2
> > or
> > > 3
> > > > > > times
> > > > > > > > more
> > > > > > > > > data per message.
> > > > > > > > >
> > > > > > > > > I'm wondering if kafka has any performance characteristics
> > that
> > > > > > differ
> > > > > > > > for
> > > > > > > > > the 2 scenarios.
> > > > > > > > >
> > > > > > > > > I like #1 because it simplifies targeted message
> consumption,
> > > and
> > > > > > > enables
> > > > > > > > > more interesting use of TopicFilter'ing.  But
I'm unsure
> > > whether
> > > > > > there
> > > > > > > > > might be performance concerns with kafka (does
it have to
> do
> > > more
> > > > > > work
> > > > > > > to
> > > > > > > > > separately manage each topic?).  Is this a common
use case,
> > or
> > > > not?
> > > > > > > > >
> > > > > > > > > Thanks for any insight.
> > > > > > > > >
> > > > > > > > > Jason
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message