kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Joel Koshy <jjko...@gmail.com>
Subject Re: Tail or stdin producer?
Date Mon, 03 Oct 2011 18:00:18 GMT
> I am totally new to Kafka and we are evaluating Kafka like Felix. I'm
> looking for a utility to pipe data from a file (or multiple files) to a
> producer (like command tail -f in linux). This is an our typical use-case. I
> saw the patch (KAFKA-130) you provided but not sure if this will work well
> without loosing data if the file (or multiple files) will be rotated.

Would "tail -F --retry" work for you? Also, if these are log files and
you are using log4j, then kafka has a log4j appender that you can use
to produce messages.

>
> I would care about plugin that will such as filter message at producer;
> modify message body in order to mark some critical kind of information, etc.
> It would be good if you provide documentation about how to write a plugin
> for Kafka.

You can use async producer callback handlers to do this sort of
filtering. I don't see an example in the distribution, but it is
pretty straightforward: see
https://svn.apache.org/repos/asf/incubator/kafka/trunk/core/src/main/scala/kafka/javaapi/producer/async/CallbackHandler.java
(The kafka log4j producer currently uses the syncproducer, so you
won't be able to use callback handlers with it).

Thanks,

Joel

>
> On Sun, Oct 2, 2011 at 9:51 PM, Jay Kreps <jay.kreps@gmail.com> wrote:
>
>> Hi,
>>
>> Currently all the patch does is take input from stdin. Can you help me
>> understand what you are looking for?
>>
>> What kind of plugin are you trying to write?
>>
>> -Jay
>>
>>
>> On Thu, Sep 29, 2011 at 10:41 PM, Bao Thai Ngo <baothaingo@gmail.com>wrote:
>>
>>> Jay,
>>>
>>> Does the patch already support log rotation (file rolling)?
>>>
>>> Also, I wonder if there is any documentation about how to write a plug-in
>>> for Kafka.
>>>
>>> Thanks,
>>> ~Thai
>>>
>>> On Mon, Sep 12, 2011 at 10:17 AM, Jay Kreps <jay.kreps@gmail.com> wrote:
>>>
>>> > I added a patch for this:
>>> > https://issues.apache.org/jira/browse/KAFKA-130
>>> >
>>> > Felix, if you are still interested in this, let me know if it works for
>>> > you.
>>> >
>>> > -Jay
>>> >
>>> > On Thu, Sep 1, 2011 at 1:22 PM, Felix Giguere Villegas <
>>> > felix.giguere@mate1inc.com> wrote:
>>> >
>>> > > Hi :)
>>> > >
>>> > > Thanks for the reply :) . I have created
>>> > > KAFKA-130<https://issues.apache.org/jira/browse/KAFKA-130>
>>> > > .
>>> > >
>>> > > I'm not sure I have respected the proper standard for JIRA issues.
>>> Please
>>> > > let me know if I haven't.
>>> > >
>>> > > --
>>> > > Felix
>>> > >
>>> > >
>>> > >
>>> > > On Thu, Sep 1, 2011 at 4:04 PM, Jun Rao <junrao@gmail.com> wrote:
>>> > >
>>> > > > Hi, Felix,
>>> > > >
>>> > > > Currently, we don't have a util to pipe data from a file to a
>>> producer.
>>> > I
>>> > > > agree that it's a very convenient tool to have. Could you open
a
>>> jira
>>> > for
>>> > > > that?
>>> > > >
>>> > > > Thanks,
>>> > > >
>>> > > > Jun
>>>
>>
>

Mime
View raw message