kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Aris Risdianto <a...@smartx.kr>
Subject Re: sFlow/NetFlow/Pcap Plugin for Kafka Producer
Date Mon, 10 Oct 2016 10:20:48 GMT
Thank you for answer Michael.

Actually, I have made a simple producer from Pcap to Kafka. Since it is not
structured, so it is difficult for further processing by a consumer. But, I
will take a look at Avro as you mentioned.

I just wondering, if there are any proper implementation for this
requirement, because I couldn't find any tool in the kafka ecosystem page.

https://cwiki.apache.org/confluence/display/KAFKA/Ecosystem


Best Regards,
Aris.


On Mon, Oct 10, 2016 at 6:55 PM, Michael Noll <michael@confluent.io> wrote:

> Aris,
>
> even today you can already use Kafka to deliver Netflow/Pcap/etc. messages,
> and people are already using it for that (I did that in previous projects
> of mine, too).
>
> Simply encode your Pcap/... messages appropriately (I'd recommend to take a
> look at Avro, which allows you to structure your data similar to e.g.
> Pcap's native format [1]), and then write the encoded messages to Kafka.
> Your downstream applications can then read the encoded messages back from
> Kafka, decode, and commence processing.
>
> That was a brief summary to get you started, feel free to take a look at
> the Apache Kafka docs at kafka.apache.org and/or ask further questions
> here.
>
> -Michael
>
>
>
>
> [1] https://wiki.wireshark.org/Development/LibpcapFileFormat
>
> On Mon, Oct 10, 2016 at 11:19 AM, Aris Risdianto <aris@smartx.kr> wrote:
>
> > ​Hello,
> >
> >
> > ​Is there any plan or implementation to use Kafka for delivering
> > sFlow/NetFlow/Pcap messages?
> >
> >
> > Best Regards,
> > Aris.
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message