kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Alexis Midon <alexis.mi...@airbnb.com.INVALID>
Subject Re: automatically consume from all topics
Date Fri, 18 Sep 2015 17:46:45 GMT
Thanks Tao.
for reference, here is the corresponding listener
https://github.com/apache/kafka/blob/9dbeb71ab258955e04b46991c1baf880b07633f4/core/src/main/scala/kafka/consumer/ZookeeperConsumerConnector.scala#L1000-L1002

On Sat, Sep 12, 2015 at 5:42 AM, tao xiao <xiaotao183@gmail.com> wrote:

> It is handled by the implementation of the high level consumer
> automatically. High level consumer internal registers an listener on ZK
> topic znode. Any changes to it including topic creation high level consumer
> will be notify and and then rebalanced topics if the new topic matches the
> regex. You can check the source code high level for reference
> On Sat, Sep 12, 2015 at 5:18 AM Alexis Midon
> <alexis.midon@airbnb.com.invalid> wrote:
>
> > When a new topic is created, I agree that the regex would remain
> unchanged
> > but how would an existing consumer be notified of the topic creation?
> > afaik there's no such notification mechanism in the High level consumer.
> >
> >
> >
> > On Thu, Sep 10, 2015 at 8:43 AM, tao xiao <xiaotao183@gmail.com> wrote:
> >
> > > You can create message streams using regex that includes all topics.
> The
> > > beauty of regex is that any new topic created will be automatically
> > > consumed as long as the name of the topic matches the regex
> > >
> > > You check the method createMessageStreamsByFilter in high level API
> > >
> > > On Thu, Sep 10, 2015 at 11:03 PM Gerrit Jansen van Vuuren <
> > > gerritjvv@gmail.com> wrote:
> > >
> > > > Hi,
> > > >
> > > > I'm not sure about the high level consumer but I maintain a kafka
> > > consumer
> > > > that can add and remove topics dynamically.
> > > >
> > > > https://github.com/gerritjvv/kafka-fast
> > > > see
> > > >
> > > >
> > >
> >
> https://github.com/gerritjvv/kafka-fast/blob/master/kafka-clj/java/kakfa_clj/core/Consumer.java
> > > > if you're using java/scala
> > > >
> > > >
> > > >
> > > > On Thu, Sep 10, 2015 at 3:48 PM, Helleren, Erik <
> > > > Erik.Helleren@cmegroup.com>
> > > > wrote:
> > > >
> > > > > So, there are several ways to do this.  Lets assume the goal is to
> > add
> > > > > more topics to the application at runtime.  And that this app is
> > > > currently
> > > > > written to be distributed via the magic of consumer groups.
> Sadly, I
> > > > > don¹t think the High level consumer is well designed for this
> > > particular
> > > > > use case.  The app would have to poll using something like the
> topic
> > > list
> > > > > script (bin/kafka-topics.sh --list Š), close the existing high
> level
> > > > > consumer on a change, and start a new one.  And then do this on all
> > the
> > > > > nodes of your application (Should be easier than doing it on just
> one
> > > > > actually).  This would result in a huge latency spike and a problem
> > > when
> > > > > it comes to migrating the state involved in your example
> expectation.
> > > > >
> > > > > The next option still requires polling, but it needs a custom FT
> and
> > > > > distribution scheme.  There might need a leader so some things only
> > > > happen
> > > > > once.  Just use the simple consumer API, and have one thread per
> > > > > partition.  The leader would have to tell a follower something like
> > > > ³Start
> > > > > listening to topic X, partition Y², which is risky and difficult
to
> > do.
> > > > >
> > > > > The simplest option, assuming that each topic is independent when
> it
> > > > comes
> > > > > to expectations, is don¹t go with a cluster.  Just have a
> > > script/watcher
> > > > > app that does the polling and then, when it detects new topics, for
> > > each
> > > > > new topic: start a new instance of your app on a new box that
> listens
> > > to
> > > > > that single topic.  It might take a few seconds to startup, but its
> > > easy
> > > > > to code, easy to maintain, and easy to understand.  Which makes
> for a
> > > > more
> > > > > resilient application.
> > > > > -Erik
> > > > >
> > > > >
> > > > >
> > > > > From:  Joris Peeters <j.peeters@wintoncapital.com>
> > > > > Reply-To:  "users@kafka.apache.org" <users@kafka.apache.org>
> > > > > Date:  Thursday, September 10, 2015 at 6:09 AM
> > > > > To:  "users@kafka.apache.org" <users@kafka.apache.org>
> > > > > Subject:  automatically consume from all topics
> > > > >
> > > > >
> > > > > Hello,
> > > > >
> > > > > Is there a simple way to set up a consumer that automatically picks
> > up
> > > > all
> > > > > the topics for all the partitions, dynamically extending its range
> as
> > > new
> > > > > topics get created?
> > > > >
> > > > > The underlying idea is that we want to have a few over-arching
> > > consumers
> > > > > (I¹m aware that¹s not great for the scalability, but that¹s not
> such
> > a
> > > > > concern at present), to
> > > > > -
> > > > > Gather various statistics, metrics, system pressure, Š and dispatch
> > to
> > > > the
> > > > > appropriate  monitoring systems,
> > > > > -
> > > > > Apply some end-to-end business-logic testing, to continuously
> assert
> > > > > certain expectations (e.g. ³if this-sort-of message arrived, then
> we
> > > > > expect that-sort-of-message to be received within this time² etc).
> > > > >
> > > > >
> > > > > I¹m sure I can piece something together that does this, but perhaps
> > it
> > > > > comes out of the box. (Couldn¹t find it, though).
> > > > > We¹re using the Java client and Kafka 8.2.1.
> > > > >
> > > > > Joris Peeters
> > > > > Developer
> > > > >
> > > > > Research and Data Technology
> > > > > T:
> > > > > +44 (0) 20 8576 5800
> > > > >
> > > > > Winton
> > > > > Grove House
> > > > > 27 Hammersmith Grove
> > > > > London W6 0NE
> > > > >
> > > > > wintoncapital.com <http://www.wintoncapital.com/>
> > > > >
> > > > >  <http://www.wintoncapital.com/>
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > Winton Capital Management Limited (³Winton²) is a limited company
> > > > > registered in England and Wales with its registered offices at 16
> Old
> > > > > Bailey, London, EC4M 7EG (Registered Company No. 3311531).
> > > > >  Winton is authorised and regulated by the Financial Conduct
> > Authority
> > > in
> > > > > the United Kingdom, registered as an investment adviser with the
US
> > > > > Securities and Exchange Commission, registered with the US
> Commodity
> > > > > Futures Trading Commission and a member of the
> > > > >  National Futures Association in the United States.
> > > > > This communication, including any attachments, is confidential and
> > may
> > > be
> > > > > privileged. This email is for use by the intended recipient only.
> If
> > > you
> > > > > receive it in error, please notify the sender and
> > > > >  delete it. You should not copy or disclose all or any part of this
> > > > email.
> > > > > This email does not constitute an offer or solicitation and nothing
> > > > > contained in this email constitutes, and should not be construed
> as,
> > > > > investment advice. Prospective investors should request offering
> > > > >  materials and consult their own advisers with respect to
> investment
> > > > > decisions and inform themselves as to applicable legal
> requirements,
> > > > > exchange control regulations and taxes in the countries of their
> > > > > citizenship, residence or domicile. Past performance is
> > > > >  not indicative of future results.
> > > > > Winton takes reasonable steps to ensure the accuracy and integrity
> of
> > > its
> > > > > communications, including emails. However Winton accepts no
> liability
> > > for
> > > > > any materials transmitted. Emails are not secure
> > > > >  and cannot be guaranteed to be error free.
> > > > >
> > > > >
> > > >
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message