kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Scott Chapman <sc...@woofplanet.com>
Subject Re: using the new logstash-kafka plugin
Date Thu, 22 Jan 2015 21:41:29 GMT
Hey Joe, with other input types (like file) one can reference things like
the path in the filter section.

Is it possible to refer to the topic_id in the filter section? I tried and
nothing obvious worked.

We are encoding a few things (like host name, and type) in the name of the
topic, and would like to grok those values out.

Let me know if anything comes to mind.

Thanks!

-Scott

On Thu Jan 22 2015 at 10:00:21 AM Joseph Lawson <jlawson@roomkey.com> wrote:

> Just trying to get everything in prior to the 1.5 release.
>
> ________________________________________
> From: Scott Chapman <scott@woofplanet.com>
> Sent: Thursday, January 22, 2015 9:32 AM
> To: users@kafka.apache.org
> Subject: Re: using the new logstash-kafka plugin
>
> Awesome, what release are you targeting? Or are you able to make updates to
> the plugin outside of kafka?
>
> On Thu Jan 22 2015 at 9:31:26 AM Joseph Lawson <jlawson@roomkey.com>
> wrote:
>
> > Scott you will have to do just one topic per input right now but multiple
> > topics per group, whitelisting and blacklisting just got merged into
> > jruby-kafka and I'm working them up the chain to my logstash-kafka and
> then
> > pass it to the logstash-input/output/-kafka plugin.
> >
> > ________________________________________
> > From: Scott Chapman <scott@woofplanet.com>
> > Sent: Wednesday, January 21, 2015 8:32 PM
> > To: users@kafka.apache.org
> > Subject: using the new logstash-kafka plugin
> >
> > We are starting to use the new logstash-kafka plugin, and I am wondering
> if
> > it is possible to read multiple topics? Or do you need to create separate
> > logstashes for each topic to parse?
> >
> > We are consuming multi-line logs from a service running on a bunch of
> > different hosts, so we address that by creating single partition topics
> for
> > our producers.
> >
> > We then want to have logstash consume them for ELK.
> >
> > Thanks in advance!
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message