kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Toni Cebrián <toni.cebr...@gmail.com>
Subject Re: Routing modifications at runtime
Date Thu, 29 Jan 2015 20:24:19 GMT
Thank you all.

I'll have a look at flume and also at akka-http  and akka-streams since the
MACs will send the data to a REST endpoint.
El 29/01/2015 16:10, "Jeff Holoman" <jholoman@cloudera.com> escribió:

> Yeah if you're into Flume you can definitely do per event
> modification/routing in an interceptor with relative ease. I don't know the
> size of the total MAC addresses to look up (or actually why a hash
> partitioning scheme wouldn't just work, but w/e I assume you have your
> reasons). There's kind of an example of doing this here:
>
>
> http://blog.cloudera.com/blog/2014/11/flafka-apache-flume-meets-apache-kafka-for-event-processing/
>
> In the example in the blog it uses HBase to read some "profile" data. You
> could sub this with whatever other store you wanted (Redis, Cassandra,
> whatever)
>
> So you'd go:
>
> Your Systems -> Kafka (raw data) -> Flume Source->Interceptor->Kafka
> Channel(Raw data to the correct topic by setting the Flume Event header)
>
> Of course you could code that all yourself too. Whatever floats your boat.
> Writing interceptors is really easy and there's quite a few examples
> around.
>
> Jeff
>
> On Thu, Jan 29, 2015 at 4:10 AM, David Morales <dmorales@stratio.com>
> wrote:
>
> > Hi Toni,
> >
> > 1. Kafka can create topics on the fly, in case you need it.
> >
> > https://kafka.apache.org/08/configuration.html
> >
> > auto.create.topics.enabletrueEnable auto creation of topic on the server.
> > If this is set to true then attempts to produce, consume, or fetch
> metadata
> > for a non-existent topic will automatically create it with the default
> > replication factor and number of partitions.
> >
> >
> >
> > 2. About topic selection based on rules/dictionary, this must be solved
> on
> > your side.
> >
> > You can use custom-code in your app or an event transport solution, like
> > Flume.
> >
> > Flume 1.6 now includes a sink for Kafka, and it already supports dynamic
> > topics (by using a preprocessor)
> >
> > https://github.com/thilinamb/flume-ng-kafka-sink
> >
> >
> >    -
> >
> >    *topic*[optional]
> >    - The topic in Kafka to which the messages will be published. If this
> >       topic is mentioned, every message will be published to the same
> > topic. If
> >       dynamic topics are required, it's possible to use a preprocessor
> > instead of
> >       a static topic. It's mandatory that either of the parameters
> *topic*
> >       or *preprocessor* is provided, because the topic cannot be null
> when
> >       publishing to Kafka. If none of these parameters are provided,
> > the messages
> >       will be published to a default topic called default-flume-topic.
> >
> >
> >
> > Regards.
> >
> >
> >
> >
> > 2015-01-29 0:16 GMT+01:00 Lakshmanan Muthuraman <lakshmanan@tokbox.com>:
> >
> > > Hi Toni,
> > >
> > > Couple of thoughts.
> > >
> > > 1. Kafka behaviour need not be changed at run time. Your producers
> which
> > > push your MAC data into kafka should know to which topic it should
> write.
> > > Your producer can be flume, log stash or it can  be your own custom
> > written
> > > java producer.
> > >
> > > As long as your producer know which topic to write, they can keep
> > creating
> > > new topics as new MAC data comes through your pipeline.
> > >
> > > On Wed, Jan 28, 2015 at 12:10 PM, Toni Cebrián <toni.cebrian@gmail.com
> >
> > > wrote:
> > >
> > > > Hi,
> > > >
> > > >     I'm starting to weight different alternatives for data ingestion
> > and
> > > > I'd like to know whether Kafka meets the problem I have.
> > > >     Say we have a set of devices each with its own MAC and then we
> > > receive
> > > > data in Kafka. There is a dictionary defined elsewhere that says each
> > MAC
> > > > to which topic must publish. So I have basically 2 questions:
> > > > New MACs keep comming and the dictionary must be updated accordingly.
> > How
> > > > could I change this Kafka behaviour during runtime?
> > > > A problem for the future. Say that dictionaries are so big that they
> > > don't
> > > > fit in memory. Are there any patterns for bookkeeping internal data
> > > > structures and how route to them?
> > > >
> > > > T.
> > > >
> > >
> >
> >
> >
> > --
> >
> > David Morales de Frías  ::  +34 607 010 411 :: @dmoralesdf
> > <https://twitter.com/dmoralesdf>
> >
> >
> > <http://www.stratio.com/>
> > Vía de las dos Castillas, 33, Ática 4, 3ª Planta
> > 28224 Pozuelo de Alarcón, Madrid
> > Tel: +34 91 828 6473 // www.stratio.com // *@stratiobd
> > <https://twitter.com/StratioBD>*
> >
>
>
>
> --
> Jeff Holoman
> Systems Engineer
> 678-612-9519
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message