kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Grant Henke <ghe...@cloudera.com>
Subject Re: message filterin or "selector"
Date Thu, 06 Aug 2015 15:10:35 GMT
The filtering logic there is topic filtering and not message filtering. The
idea is to subscribe to multiple topics via a regex whitelist or black
list. This does exist today as it does not depend on understanding the
content of the message, but I don't think it is what you are looking for.

As far as message filtering goes; As Gwen said, "The way Kafka is currently
implemented is that Kafka is not aware of the content of messages, so there
is no Selector logic available." However, If you know upfront how you would
like to filter the messages you could write your producer to use multiple
topics, or even some custom partitioning. And implement a consumer that can
understand and filter based on that logic. However, that would be an
involved and creative implementation based on your use case.

I would recommend starting simple and just dropping the messages you don't
care about on the consumer side. If throughput becomes a problem, then
consider alternatives.


On Thu, Aug 6, 2015 at 9:47 AM, Alvaro Gareppe <agareppe@gmail.com> wrote:

> Is this implemented ?
> https://cwiki.apache.org/confluence/display/KAFKA/Consumer+API+changes ?
>
> This is message filtering on the client or server side ?
>
> On Tue, Aug 4, 2015 at 9:54 PM, Gwen Shapira <gwen@confluent.io> wrote:
>
> > The way Kafka is currently implemented is that Kafka is not aware of the
> > content of messages, so there is no Selector logic available.
> >
> > The way to go is to implement the Selector in your client - i.e. your
> > consume() loop will get all messages but will throw away those that don't
> > fit your pattern.
> >
> >
> > It may be worthwhile to add a ticket for pluggable selector logic in the
> > new consumer. I can't guarantee it will happen, there are infinite things
> > that can be plugged into consumers and we need to draw the line
> somewhere,
> > but worth a discussion.
> >
> > On Tue, Aug 4, 2015 at 2:05 PM, Alvaro Gareppe <agareppe@gmail.com>
> wrote:
> >
> > > The is way to implement a "selector" logic in kafka (similar to JMS
> > > selectors)
> > >
> > > So, allow to consume a message if only the message contains certain
> > header
> > > or content ?
> > >
> > > I'm evaluating to migrate from ActiveMQ to kafka and I'm using the
> > selector
> > > logic widely in the application
> > >
> > > --
> > > Ing. Alvaro Gareppe
> > > agareppe@gmail.com
> > >
> >
>
>
>
> --
> Ing. Alvaro Gareppe
> agareppe@gmail.com
>



-- 
Grant Henke
Software Engineer | Cloudera
grant@cloudera.com | twitter.com/gchenke | linkedin.com/in/granthenke

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message