metron-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Otto Fowler <ottobackwa...@gmail.com>
Subject Re: Develop some parsers
Date Fri, 07 Apr 2017 16:07:12 GMT
Hopefully as we make producing parsers outside of metron easier ( if not
possible ) we will be able to get more traction around this.


On April 7, 2017 at 10:54:55, Mark de Rijk (me.derijk@gmail.com) wrote:

I do have datasets for some parsers separately such as for Infoblox.
Can we centralize the parser development somewhere as a document so for
instance we can vote on which ones the project needs first?
I would be happy to work on parsers and coordinate the retrieval of data
sets, anonymization/obfuscation and processing/generating the parsers.
Once I am more comfortable in Java and such I can then help with more
stuff.

Thanks for the welcome and I hope to start generating added value as soon
as possible.


On 07/April/2017 15:44 , "Nick Allen" <nick@nickallen.org> wrote:

I think the primary limitation for parser contributors is that you need a
fair amount of good data to work with. Building a parser from a small
amount of test data will probably miss some corner cases that one would see
"in the wild." I don't think there are any right or wrong answers here
though. If you have access to good Infoblox data, for example, then start
there.

I would also suggest that you try building a parser with Grok expressions
first. You can do quite a bit with grok expressions. Get a feel for what
you can do with those, what the limitations are, and then transition into
Java parsers only as you see a need.

Really glad that you are joining the community. I look forward to your
contributions.




On Fri, Apr 7, 2017 at 10:33 AM, Mark de Rijk <me.derijk@gmail.com> wrote:

> Ali,
>
> Coding wise I am still getting my footing on and doing a Java course
> online.
> So building in unit tests I am afraid is a bit further away from me.
> For the parsers:
>
> - Infoblox Syslog
> - Cisco IOS Syslog
> - Unix Syslog
> - Alcatel Syslog
> - Sendmail Syslog
> - Blackberry Enterprise Server File
>
> I have also developed file and DB based connectors but as said I need to
> figure it out how to actually develop parsers for the Metron platform
> specifically.
>
> Is there a list of most requested devices documented somewhere so I can
> focus my efforts?
>
> Cheers,
> Mark
>
>
> On Fri, Apr 7, 2017 at 2:56 PM, Ali Nazemian <alinazemian@gmail.com>
> wrote:
>
> > Mark,
> >
> > Can you please specify the Parsers you are familiar with? We are
> > prioritizing the parser implementation, so your effort can help us to
> > target them faster.
> >
> > Cheers,
> > Ali
> >
> > On Fri, Apr 7, 2017 at 11:41 PM, Mark De Rijk <me.derijk@gmail.com>
> wrote:
> >
> > > I will review them this afternoon. Thanks for that.
> > >
> > > Sent from my iPhone
> > >
> > > > On 7 Apr 2017, at 14:37, Ali Nazemian <alinazemian@gmail.com>
wrote:
> > > >
> > > > Mark,
> > > >
> > > > Have you seen the following pages?
> > > >
> > > > https://cwiki.apache.org/confluence/display/METRON/
> > > Development+Guidelines
> > > > https://cwiki.apache.org/confluence/display/METRON/
> Metron+Development+
> > > Environment+Setup+Instructions
> > > > https://cwiki.apache.org/confluence/display/METRON/
> Community+Resources
> > > >
> > > >
> > > >> On Fri, Apr 7, 2017 at 11:20 PM, Mark de Rijk <me.derijk@gmail.com>

> > > wrote:
> > > >>
> > > >> To clarify I have written a lot of parsers for ArcSight over the
> years
> > > and
> > > >> I would like to start contributing by developing parsers for the
> > Metron
> > > >> project.
> > > >> Is there any documentation that will help me get started so I can
> > start
> > > >> cranking them out?
> > > >> This is the first open source project I am looking to contribute
to
> so
> > > >> forgive me If I am asking stupid questions.
> > > >>
> > > >>
> > > >>
> > > >> On Fri, Apr 7, 2017 at 2:13 PM, Otto Fowler <
> ottobackwards@gmail.com>
> > > >> wrote:
> > > >>
> > > >>> I also believe that grok parsers can be added through
configuration
> > > only,
> > > >>> without having to
> > > >>> compile a parser.
> > > >>>
> > > >>> You can add a parser configuration targeting the basic grok
parser
> > and
> > > >> just
> > > >>> provide the grok
> > > >>> parser rules.
> > > >>>
> > > >>>
> > > >>> Just as a heads up, I’m currently working on the parsers to
allow
> for
> > > >>> writing and maintaining parsers
> > > >>> outside the metron code tree, including providing a maven
> archetype.
> > > >> This
> > > >>> will allow you to create parsers
> > > >>> without having to maintain a fork etc.
> > > >>>
> > > >>> Keep an eye out for METRON-258 as a PR on the list.
> > > >>>
> > > >>>
> > > >>>
> > > >>> On April 7, 2017 at 08:54:35, Justin Leet (justinjleet@gmail.com)
> > > wrote:
> > > >>>
> > > >>> My understanding of Grok vs Java is to provide a tradeoff for
ease
> of
> > > >>> implementation vs performance (plus Java can also handle parsing
> that
> > > >> would
> > > >>> be too complicated for Grok.
> > > >>>
> > > >>> Grok is less performant and handles less complex parsing, but
it's
> > easy
> > > >> to
> > > >>> get things going and potentially maintained without writing and
> > > compiling
> > > >>> Java.
> > > >>>
> > > >>> The Java implementation will be better for performance and can
> handle
> > > >> more
> > > >>> complicated parsing Grok can't.
> > > >>>
> > > >>> I believe the preference has generally been for Grok parsers if
> > > >>> appropriate, otherwise Java parsers.
> > > >>>
> > > >>> Justin
> > > >>>
> > > >>> On Fri, Apr 7, 2017 at 8:09 AM, Ali Nazemian <
> alinazemian@gmail.com>
> > > >>> wrote:
> > > >>>
> > > >>>> Hi Mark,
> > > >>>>
> > > >>>> Yeah, that would be great. Can you please specify which devices
> you
> > > >> have
> > > >>>> developed so far?
> > > >>>>
> > > >>>> Cheers,
> > > >>>> Ali
> > > >>>>
> > > >>>> On Fri, Apr 7, 2017 at 4:10 PM, Mark De Rijk <me.derijk@gmail.com
> >
> > > >>> wrote:
> > > >>>>
> > > >>>>> Dear all,
> > > >>>>>
> > > >>>>> I am a heavy arcsight user and I have written quite a
few
parsers
> > > >> over
> > > >>>>> time.
> > > >>>>> I am new to contributing to open source projects however.
> > > >>>>> @Ali, would you like to cooperate on development of some
parsers?
> > > >>>>>
> > > >>>>> Kind Regards,
> > > >>>>> Mark de Rijk
> > > >>>>>
> > > >>>>>
> > > >>>>>> On 7 Apr 2017, at 04:30, Ali Nazemian <alinazemian@gmail.com>
> > > >> wrote:
> > > >>>>>>
> > > >>>>>> Hi all,
> > > >>>>>>
> > > >>>>>> We are going to develop some parsers and have some
contribution
> to
> > > >>> the
> > > >>>>>> community as a start point. I was wondering what the
reason is
> > > >> behind
> > > >>>>>> choosing Grok statements for some of the implementations
and
> Java
> > > >>> regex
> > > >>>>> for
> > > >>>>>> other ones? Is there any policy for that? Probably
it would be
> > > >> better
> > > >>>> to
> > > >>>>>> have the Java regex implementation due to performance
concerns.
> > > >>>> However,
> > > >>>>> I
> > > >>>>>> am sure there is a reason that some of them have been
> implemented
> > > >>> with
> > > >>>>>> using Grok statements.
> > > >>>>>>
> > > >>>>>> Regards,
> > > >>>>>> Ali
> > > >>>>>
> > > >>>>
> > > >>>>
> > > >>>>
> > > >>>> --
> > > >>>> A.Nazemian
> > > >>>>
> > > >>>
> > > >>
> > > >
> > > >
> > > >
> > > > --
> > > > A.Nazemian
> > >
> >
> >
> >
> > --
> > A.Nazemian
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message