nifi-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Matt Burgess <mattyb...@apache.org>
Subject Re: Regular Expression, or Similar Processor
Date Tue, 17 Jan 2017 23:35:40 GMT
Oziel,

The ListenSyslog processor [1] will apply the regular expression and
extract each of the fields into flow file attributes. From there you
could use AttributesToJSON [2] with "Include Core Attributes" set to
false, that should give you fields named "syslog.hostname" for
example. You could use JoltTransformJSON [3] if you need to
rename/reorganize the fields, or if you need more complex logic and
are familiar with a scripting language such as Groovy, Javascript,
Python, Ruby, or Lua, you could use ExecuteScript [4] to build a
custom JSON output, I have examples of custom JSON transformations
using Groovy [5], Javascript [6], and Jython [7].

If you are looking for the regular expression(s) to do the parsing
yourself (perhaps with ExtractText [8]), you can find them here [9].

Regards,
Matt

[1] https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi.processors.standard.ListenSyslog/index.html
[2] https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi.processors.standard.AttributesToJSON/index.html
[3] https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi.processors.standard.JoltTransformJSON/index.html
[4] https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi.processors.script.ExecuteScript/index.html
[5] http://funnifi.blogspot.com/2016/02/executescript-json-to-json-conversion.html
[6] http://funnifi.blogspot.com/2016/03/executescript-json-to-json-revisited.html
[7] http://funnifi.blogspot.com/2016/03/executescript-json-to-json-revisited_14.html
[8] https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi.processors.standard.ExtractText/index.html
[9] https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/syslog/SyslogParser.java#L36

On Tue, Jan 17, 2017 at 5:28 PM, Hinojosa, Ozzie
<Oziel.Hinojosa@chevron.com> wrote:
> Hi,
>
> I am in the process of reading the Apache NiFi documentation and setting up a single
instance to obtain basic working knowledge of the platform. But in the meantime I need an
answer regarding Syslog messages and guidance on which educational resources would be best
to leverage in order to understand NiFi in depth and in the shortest time possible.
>
> My questions regarding Syslog. How can I parse the FlowFile contents produce by a ListenSyslog
processor. By parse, I mean apply regular expressions and transform the entire contents to
JSON. I know extensibility is an integral part of the platform, but again I am new to NiFi
and I am looking to confirm the feasibility of parsing Syslog messages; to add additional
complexity Netflows would have the same requirement as the Syslog messages. What is available
out of the box for Netflows and JFlows?
>
> Thanks, much appreciated.
>
> Oziel Hinojosa
> Security Technologies Analyst
> Strategy, Service Delivery & Programs (SSDP)
> oziel.hinojosa@chevron.com
>
> CITC - IRSM
> 1600 Smith St.  HOU160-23042B
> Houston, TX 77002
>
> Tel +1 713 754 4749
>
> Use http://csocbehavior.chevron.com/ to give me or others CSOC Behavioral Feedback, my
CAI is OZNH.
>

Mime
View raw message