kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From navneet sharma <navneetsharma0...@gmail.com>
Subject Re: Dynamic weblog processing
Date Mon, 09 Apr 2012 04:04:45 GMT
Thanks a lot for your responses.

I think i can start with prducer-in-app approach.

I saw the logger option as well- Kafka appender. Will it be more faster and
efficient? Anybody tried that?



On Fri, Apr 6, 2012 at 12:58 AM, Felix GV <felix@mate1inc.com> wrote:

> As Hisham mentioned, what I've been working on is your option #2, and that
> can be done by using the Kafka APIs...
>
> Currently, the easiest way to get this up and running quickly would
> probably be your option #1, using the kafka-console-producer that was added
> by the kind Kafka folks in KAFKA-130 :)
>
> (It's available in 0.7)
>
> --
> Felix
>
>
>
> On Wed, Apr 4, 2012 at 9:56 PM, Hisham Mardam-Bey <hisham@mate1inc.com
> >wrote:
>
> > On Wed, Apr 4, 2012 at 5:20 AM, navneet sharma
> > <navneetsharma0505@gmail.com> wrote:
> > > Hi,
> > >
> > > I am new to kafka. If i have an application which is generating logs
> and
> > i
> > > need to read and do some CEP on these logs.
> > >
> > > For CEP, we are planning to use Esper and for reading weblogs, we are
> > > examining Kafka. I have few questions:
> > > 1) What will be a producer in this case. Will it be an external
> > > entity(standalone program) which will do file I/o and send weblogs to
> > kafka
> > > broker?
> > > 2) Or will it be part of application itself, so that it can read logs
> in
> > > parallel with the logging utility?
> > >
> > > Any pointers will be helpful...
> > >
> > > Thanks,
> > > Navneet
> >
> > Navneet,
> >
> > We're currently doing that for our Tomcat servers. We've built an
> > access log valve (inspired by the one bundled with Tomcat that writes
> > log files to disk) that will format logs and push them into Kafka. We
> > then pull the logs and push them into HBase for analysis. FelixGV (on
> > this list) is implementing the code if you have questions for him.
> >
> > Hope this helps,
> >
> > hisham.
> >
> > --
> > Hisham Mardam Bey
> >
> > A: Because it messes up the order in which people normally read text.
> > Q: Why is top-posting such a bad thing?
> > A: Top-posting.
> > Q: What is the most annoying thing in e-mail?
> >
> > -=[ Codito Ergo Sum ]=-
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message