kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Hisham Mardam-Bey <his...@mate1inc.com>
Subject Re: Dynamic weblog processing
Date Thu, 05 Apr 2012 01:56:09 GMT
On Wed, Apr 4, 2012 at 5:20 AM, navneet sharma
<navneetsharma0505@gmail.com> wrote:
> Hi,
>
> I am new to kafka. If i have an application which is generating logs and i
> need to read and do some CEP on these logs.
>
> For CEP, we are planning to use Esper and for reading weblogs, we are
> examining Kafka. I have few questions:
> 1) What will be a producer in this case. Will it be an external
> entity(standalone program) which will do file I/o and send weblogs to kafka
> broker?
> 2) Or will it be part of application itself, so that it can read logs in
> parallel with the logging utility?
>
> Any pointers will be helpful...
>
> Thanks,
> Navneet

Navneet,

We're currently doing that for our Tomcat servers. We've built an
access log valve (inspired by the one bundled with Tomcat that writes
log files to disk) that will format logs and push them into Kafka. We
then pull the logs and push them into HBase for analysis. FelixGV (on
this list) is implementing the code if you have questions for him.

Hope this helps,

hisham.

-- 
Hisham Mardam Bey

A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?
A: Top-posting.
Q: What is the most annoying thing in e-mail?

-=[ Codito Ergo Sum ]=-

Mime
View raw message