kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Felix GV <fe...@mate1inc.com>
Subject Re: Dynamic weblog processing
Date Thu, 05 Apr 2012 19:28:23 GMT
As Hisham mentioned, what I've been working on is your option #2, and that
can be done by using the Kafka APIs...

Currently, the easiest way to get this up and running quickly would
probably be your option #1, using the kafka-console-producer that was added
by the kind Kafka folks in KAFKA-130 :)

(It's available in 0.7)

--
Felix



On Wed, Apr 4, 2012 at 9:56 PM, Hisham Mardam-Bey <hisham@mate1inc.com>wrote:

> On Wed, Apr 4, 2012 at 5:20 AM, navneet sharma
> <navneetsharma0505@gmail.com> wrote:
> > Hi,
> >
> > I am new to kafka. If i have an application which is generating logs and
> i
> > need to read and do some CEP on these logs.
> >
> > For CEP, we are planning to use Esper and for reading weblogs, we are
> > examining Kafka. I have few questions:
> > 1) What will be a producer in this case. Will it be an external
> > entity(standalone program) which will do file I/o and send weblogs to
> kafka
> > broker?
> > 2) Or will it be part of application itself, so that it can read logs in
> > parallel with the logging utility?
> >
> > Any pointers will be helpful...
> >
> > Thanks,
> > Navneet
>
> Navneet,
>
> We're currently doing that for our Tomcat servers. We've built an
> access log valve (inspired by the one bundled with Tomcat that writes
> log files to disk) that will format logs and push them into Kafka. We
> then pull the logs and push them into HBase for analysis. FelixGV (on
> this list) is implementing the code if you have questions for him.
>
> Hope this helps,
>
> hisham.
>
> --
> Hisham Mardam Bey
>
> A: Because it messes up the order in which people normally read text.
> Q: Why is top-posting such a bad thing?
> A: Top-posting.
> Q: What is the most annoying thing in e-mail?
>
> -=[ Codito Ergo Sum ]=-
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message