nifi-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Bryan Bende <bbe...@gmail.com>
Subject Re: Regular Expression, or Similar Processor
Date Tue, 17 Jan 2017 23:45:36 GMT
One more resource to add to Matt's list:

https://blogs.apache.org/nifi/entry/storing_syslog_events_in_hbase

Thanks,

Bryan

On Tue, Jan 17, 2017 at 6:35 PM, Matt Burgess <mattyb149@apache.org> wrote:

> Oziel,
>
> The ListenSyslog processor [1] will apply the regular expression and
> extract each of the fields into flow file attributes. From there you
> could use AttributesToJSON [2] with "Include Core Attributes" set to
> false, that should give you fields named "syslog.hostname" for
> example. You could use JoltTransformJSON [3] if you need to
> rename/reorganize the fields, or if you need more complex logic and
> are familiar with a scripting language such as Groovy, Javascript,
> Python, Ruby, or Lua, you could use ExecuteScript [4] to build a
> custom JSON output, I have examples of custom JSON transformations
> using Groovy [5], Javascript [6], and Jython [7].
>
> If you are looking for the regular expression(s) to do the parsing
> yourself (perhaps with ExtractText [8]), you can find them here [9].
>
> Regards,
> Matt
>
> [1] https://nifi.apache.org/docs/nifi-docs/components/org.
> apache.nifi.processors.standard.ListenSyslog/index.html
> [2] https://nifi.apache.org/docs/nifi-docs/components/org.
> apache.nifi.processors.standard.AttributesToJSON/index.html
> [3] https://nifi.apache.org/docs/nifi-docs/components/org.
> apache.nifi.processors.standard.JoltTransformJSON/index.html
> [4] https://nifi.apache.org/docs/nifi-docs/components/org.
> apache.nifi.processors.script.ExecuteScript/index.html
> [5] http://funnifi.blogspot.com/2016/02/executescript-json-to-
> json-conversion.html
> [6] http://funnifi.blogspot.com/2016/03/executescript-json-to-
> json-revisited.html
> [7] http://funnifi.blogspot.com/2016/03/executescript-json-to-
> json-revisited_14.html
> [8] https://nifi.apache.org/docs/nifi-docs/components/org.
> apache.nifi.processors.standard.ExtractText/index.html
> [9] https://github.com/apache/nifi/blob/master/nifi-nar-
> bundles/nifi-standard-bundle/nifi-standard-processors/src/
> main/java/org/apache/nifi/processors/standard/syslog/SyslogParser.java#L36
>
> On Tue, Jan 17, 2017 at 5:28 PM, Hinojosa, Ozzie
> <Oziel.Hinojosa@chevron.com> wrote:
> > Hi,
> >
> > I am in the process of reading the Apache NiFi documentation and setting
> up a single instance to obtain basic working knowledge of the platform. But
> in the meantime I need an answer regarding Syslog messages and guidance on
> which educational resources would be best to leverage in order to
> understand NiFi in depth and in the shortest time possible.
> >
> > My questions regarding Syslog. How can I parse the FlowFile contents
> produce by a ListenSyslog processor. By parse, I mean apply regular
> expressions and transform the entire contents to JSON. I know extensibility
> is an integral part of the platform, but again I am new to NiFi and I am
> looking to confirm the feasibility of parsing Syslog messages; to add
> additional complexity Netflows would have the same requirement as the
> Syslog messages. What is available out of the box for Netflows and JFlows?
> >
> > Thanks, much appreciated.
> >
> > Oziel Hinojosa
> > Security Technologies Analyst
> > Strategy, Service Delivery & Programs (SSDP)
> > oziel.hinojosa@chevron.com
> >
> > CITC - IRSM
> > 1600 Smith St.  HOU160-23042B
> > Houston, TX 77002
> >
> > Tel +1 713 754 4749
> >
> > Use http://csocbehavior.chevron.com/ to give me or others CSOC
> Behavioral Feedback, my CAI is OZNH.
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message