kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Andrew Otto <o...@wikimedia.org>
Subject Re: Filter plugins in Kafka
Date Mon, 02 May 2016 13:43:21 GMT
If you want something really simple and hacky, you could use kafkatee[1]
and kafkacat[2] together:

kafkatee.conf:

input [encoding=string] pipe tail -f a.log
output pipe 1 grep -v ’not this’ | kafkacat -P -b b1:9092 -t mytopic

[1] https://github.com/wikimedia/analytics-kafkatee
[2] https://github.com/edenhill/kafkacat

On Fri, Apr 29, 2016 at 12:24 PM, Rad Gruchalski <radek@gruchalski.com>
wrote:

> Apache Samza is the way to go. Never used Kafka Streams so no opinion on
> that one.
>
>
>
>
>
>
>
>
>
>
> Best regards,
> Radek Gruchalski
> radek@gruchalski.com (mailto:radek@gruchalski.com) (mailto:
> radek@gruchalski.com)
> de.linkedin.com/in/radgruchalski/ (
> http://de.linkedin.com/in/radgruchalski/)
>
> Confidentiality:
> This communication is intended for the above-named person and may be
> confidential and/or legally privileged.
> If it has come to you in error you must take no action based on it, nor
> must you copy or show it to anyone; please delete/destroy and inform the
> sender immediately.
>
>
>
> On Friday, 29 April 2016 at 13:51, Subramanian Karunanithi wrote:
>
> > Hi,
> >
> > When a stream of data passes through Kafka, wanted to apply the filter
> and
> > then let that message pass through to partitions.
> >
> > Regards,
> > Subramanian. K
> > On Apr 26, 2016 12:33, "Marko Bonaći" <marko.bonaci@sematext.com
> (mailto:marko.bonaci@sematext.com)> wrote:
> >
> > > Instantly reminded me of Streams API, where you can use Java8 streams
> > > semantics (filter being one of them) to do the first thing in
> Gouzhang's
> > > response (filter messages from one topic into another - I assume that's
> > > what you were looking for).
> > >
> > > Marko Bonaći
> > > Monitoring | Alerting | Anomaly Detection | Centralized Log Management
> > > Solr & Elasticsearch Support
> > > Sematext <http://sematext.com/> | Contact
> > > <http://sematext.com/about/contact.html>
> > >
> > > On Tue, Apr 26, 2016 at 6:22 PM, Guozhang Wang <wangguoz@gmail.com
> (mailto:wangguoz@gmail.com)> wrote:
> > >
> > > > Hi Subramanian,
> > > >
> > > > Could you elaborate a bit more on "filtering"? Do you want to read
> raw
> > > data
> > > > from Kafka, and send the filtered data back to Kafka as a separate
> topic,
> > > > or do you want to read raw data from an external service and send the
> > > > filtered data into Kafka?
> > > >
> > > >
> > > > Guozhang
> > > >
> > > >
> > > > On Tue, Apr 26, 2016 at 8:07 AM, Subramanian Karunanithi <
> > > > sububack@gmail.com (mailto:sububack@gmail.com)
> > > > > wrote:
> > > >
> > > >
> > > > > Hi,
> > > > >
> > > > > Do we have any plugin which is available, which can be used as a
> > > > filtering
> > > > > mechanism on the data it's working on?
> > > > >
> > > > > Regards,
> > > > > Subramanian. K
> > > > >
> > > >
> > > >
> > > >
> > > >
> > > > --
> > > > -- Guozhang
> > > >
> > >
> > >
> >
> >
> >
>
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message