kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Bao Thai Ngo <baothai...@gmail.com>
Subject Re: Tail or stdin producer?
Date Tue, 04 Oct 2011 03:08:42 GMT
Thanks for the reply Joel. Please see my answer below

On Tue, Oct 4, 2011 at 1:00 AM, Joel Koshy <jjkoshy@gmail.com> wrote:

> > I am totally new to Kafka and we are evaluating Kafka like Felix. I'm
> > looking for a utility to pipe data from a file (or multiple files) to a
> > producer (like command tail -f in linux). This is an our typical
> use-case. I
> > saw the patch (KAFKA-130) you provided but not sure if this will work
> well
> > without loosing data if the file (or multiple files) will be rotated.
>
> Would "tail -F --retry" work for you? Also, if these are log files and
> you are using log4j, then kafka has a log4j appender that you can use
> to produce messages.
>
>
I believe tail would work but will not be convenient when people have
windows box (not always able to use similar utility for windows).

Also, log files are produced from variety of services including non-java
applications


> >
> > I would care about plugin that will such as filter message at producer;
> > modify message body in order to mark some critical kind of information,
> etc.
> > It would be good if you provide documentation about how to write a plugin
> > for Kafka.
>
> You can use async producer callback handlers to do this sort of
> filtering. I don't see an example in the distribution, but it is
> pretty straightforward: see
>
> https://svn.apache.org/repos/asf/incubator/kafka/trunk/core/src/main/scala/kafka/javaapi/producer/async/CallbackHandler.java
> (The kafka log4j producer currently uses the syncproducer, so you
> won't be able to use callback handlers with it).
>
> great ! I will look at this.

BTW, do you know is there any documentation like user guide or something
similar? Quickstart from Kafka does not help much.


> Thanks,
>
> Joel
>
> Thanks,
~Thai


>  >
> > On Sun, Oct 2, 2011 at 9:51 PM, Jay Kreps <jay.kreps@gmail.com> wrote:
> >
> >> Hi,
> >>
> >> Currently all the patch does is take input from stdin. Can you help me
> >> understand what you are looking for?
> >>
> >> What kind of plugin are you trying to write?
> >>
> >> -Jay
> >>
> >>
> >> On Thu, Sep 29, 2011 at 10:41 PM, Bao Thai Ngo <baothaingo@gmail.com
> >wrote:
> >>
> >>> Jay,
> >>>
> >>> Does the patch already support log rotation (file rolling)?
> >>>
> >>> Also, I wonder if there is any documentation about how to write a
> plug-in
> >>> for Kafka.
> >>>
> >>> Thanks,
> >>> ~Thai
> >>>
> >>> On Mon, Sep 12, 2011 at 10:17 AM, Jay Kreps <jay.kreps@gmail.com>
> wrote:
> >>>
> >>> > I added a patch for this:
> >>> > https://issues.apache.org/jira/browse/KAFKA-130
> >>> >
> >>> > Felix, if you are still interested in this, let me know if it works
> for
> >>> > you.
> >>> >
> >>> > -Jay
> >>> >
> >>> > On Thu, Sep 1, 2011 at 1:22 PM, Felix Giguere Villegas <
> >>> > felix.giguere@mate1inc.com> wrote:
> >>> >
> >>> > > Hi :)
> >>> > >
> >>> > > Thanks for the reply :) . I have created
> >>> > > KAFKA-130<https://issues.apache.org/jira/browse/KAFKA-130>
> >>> > > .
> >>> > >
> >>> > > I'm not sure I have respected the proper standard for JIRA issues.
> >>> Please
> >>> > > let me know if I haven't.
> >>> > >
> >>> > > --
> >>> > > Felix
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message