kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From David Birdsong <david.birds...@gmail.com>
Subject Re: Suggestions of pulling local application logs into Kafka
Date Tue, 03 May 2016 21:07:18 GMT
On Tue, May 3, 2016 at 1:00 PM Banias H <banias4spark@gmail.com> wrote:

> I should add Flume is not an option for various reasons.
>
> On Tue, May 3, 2016 at 2:49 PM, Banias H <banias4spark@gmail.com> wrote:
>
> > We use Kafka (0.9.x) internally in our pipeline and now we would like to
> > ingest application logs sitting in local file system of servers external
> to
> > the Kafka cluster.
> >
> > We could write a producer program running on the application servers to
> > push files to Kafka. However we wonder if we can leverage Kafka Connect
> to
> > pull files into Kafka. It requires a connector to access local file
> systems
> > and I am not sure if I can make use of existing connectors or I need to
> > write one.
> >
> > Any thoughts or reference would be helpful.
>

heka - great at reading arbitrary inputs, files being a first-class input,
can write to kafka. next release should see use of librdkafka which should
handle leader changeovers more smoothly than the existing go library.



> >
> > Thanks,
> > B
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message