metron-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mark de Rijk <me.der...@gmail.com>
Subject Re: Develop some parsers
Date Fri, 07 Apr 2017 14:54:45 GMT
I do have datasets for some parsers separately such as for Infoblox.
Can we centralize the parser development somewhere as a document so for instance we can vote
on which ones the project needs first?
I would be happy to work on parsers and coordinate the retrieval of data sets, anonymization/obfuscation
and processing/generating the parsers.
Once I am more comfortable in Java and such I can then help with more stuff.

Thanks for the welcome and I hope to start generating added value as soon as possible.


On 07/April/2017 15:44 , "Nick Allen" <nick@nickallen.org> wrote:

    I think the primary limitation for parser contributors is that you need a
    fair amount of good data to work with.  Building a parser from a small
    amount of test data will probably miss some corner cases that one would see
    "in the wild."   I don't think there are any right or wrong answers here
    though.  If you have access to good Infoblox data, for example, then start
    there.
    
    I would also suggest that you try building a parser with Grok expressions
    first.  You can do quite a bit with grok expressions.  Get a feel for what
    you can do with those, what the limitations are, and then transition into
    Java parsers only as you see a need.
    
    Really glad that you are joining the community.  I look forward to your
    contributions.
    
    
    
    
    On Fri, Apr 7, 2017 at 10:33 AM, Mark de Rijk <me.derijk@gmail.com> wrote:
    
    > Ali,
    >
    > Coding wise I am still getting my footing on and doing a Java course
    > online.
    > So building in unit tests I am afraid is a bit further away from me.
    > For the parsers:
    >
    > - Infoblox Syslog
    > - Cisco IOS Syslog
    > - Unix Syslog
    > - Alcatel Syslog
    > - Sendmail Syslog
    > - Blackberry Enterprise Server File
    >
    > I have also developed file and DB based connectors but as said I need to
    > figure it out how to actually develop parsers for the Metron platform
    > specifically.
    >
    > Is there a list of most requested devices documented somewhere so I can
    > focus my efforts?
    >
    > Cheers,
    > Mark
    >
    >
    > On Fri, Apr 7, 2017 at 2:56 PM, Ali Nazemian <alinazemian@gmail.com>
    > wrote:
    >
    > > Mark,
    > >
    > > Can you please specify the Parsers you are familiar with? We are
    > > prioritizing the parser implementation, so your effort can help us to
    > > target them faster.
    > >
    > > Cheers,
    > > Ali
    > >
    > > On Fri, Apr 7, 2017 at 11:41 PM, Mark De Rijk <me.derijk@gmail.com>
    > wrote:
    > >
    > > > I will review them this afternoon. Thanks for that.
    > > >
    > > > Sent from my iPhone
    > > >
    > > > > On 7 Apr 2017, at 14:37, Ali Nazemian <alinazemian@gmail.com>
wrote:
    > > > >
    > > > > Mark,
    > > > >
    > > > > Have you seen the following pages?
    > > > >
    > > > > https://cwiki.apache.org/confluence/display/METRON/
    > > > Development+Guidelines
    > > > > https://cwiki.apache.org/confluence/display/METRON/
    > Metron+Development+
    > > > Environment+Setup+Instructions
    > > > > https://cwiki.apache.org/confluence/display/METRON/
    > Community+Resources
    > > > >
    > > > >
    > > > >> On Fri, Apr 7, 2017 at 11:20 PM, Mark de Rijk <me.derijk@gmail.com>
    > > > wrote:
    > > > >>
    > > > >> To clarify I have written a lot of parsers for ArcSight over the
    > years
    > > > and
    > > > >> I would like to start contributing by developing parsers for the
    > > Metron
    > > > >> project.
    > > > >> Is there any documentation that will help me get started so I
can
    > > start
    > > > >> cranking them out?
    > > > >> This is the first open source project I am looking to contribute
to
    > so
    > > > >> forgive me If I am asking stupid questions.
    > > > >>
    > > > >>
    > > > >>
    > > > >> On Fri, Apr 7, 2017 at 2:13 PM, Otto Fowler <
    > ottobackwards@gmail.com>
    > > > >> wrote:
    > > > >>
    > > > >>> I also believe that grok parsers can be added through configuration
    > > > only,
    > > > >>> without having to
    > > > >>> compile a parser.
    > > > >>>
    > > > >>> You can add a parser configuration targeting the basic grok
parser
    > > and
    > > > >> just
    > > > >>> provide the grok
    > > > >>> parser rules.
    > > > >>>
    > > > >>>
    > > > >>> Just as a heads up, I’m currently working on the parsers
to allow
    > for
    > > > >>> writing and maintaining parsers
    > > > >>> outside the metron code tree, including providing a maven
    > archetype.
    > > > >> This
    > > > >>> will allow you to create parsers
    > > > >>> without having to maintain a fork etc.
    > > > >>>
    > > > >>> Keep an eye out for METRON-258 as a PR on the list.
    > > > >>>
    > > > >>>
    > > > >>>
    > > > >>> On April 7, 2017 at 08:54:35, Justin Leet (justinjleet@gmail.com)
    > > > wrote:
    > > > >>>
    > > > >>> My understanding of Grok vs Java is to provide a tradeoff
for ease
    > of
    > > > >>> implementation vs performance (plus Java can also handle parsing
    > that
    > > > >> would
    > > > >>> be too complicated for Grok.
    > > > >>>
    > > > >>> Grok is less performant and handles less complex parsing,
but it's
    > > easy
    > > > >> to
    > > > >>> get things going and potentially maintained without writing
and
    > > > compiling
    > > > >>> Java.
    > > > >>>
    > > > >>> The Java implementation will be better for performance and
can
    > handle
    > > > >> more
    > > > >>> complicated parsing Grok can't.
    > > > >>>
    > > > >>> I believe the preference has generally been for Grok parsers
if
    > > > >>> appropriate, otherwise Java parsers.
    > > > >>>
    > > > >>> Justin
    > > > >>>
    > > > >>> On Fri, Apr 7, 2017 at 8:09 AM, Ali Nazemian <
    > alinazemian@gmail.com>
    > > > >>> wrote:
    > > > >>>
    > > > >>>> Hi Mark,
    > > > >>>>
    > > > >>>> Yeah, that would be great. Can you please specify which
devices
    > you
    > > > >> have
    > > > >>>> developed so far?
    > > > >>>>
    > > > >>>> Cheers,
    > > > >>>> Ali
    > > > >>>>
    > > > >>>> On Fri, Apr 7, 2017 at 4:10 PM, Mark De Rijk <me.derijk@gmail.com
    > >
    > > > >>> wrote:
    > > > >>>>
    > > > >>>>> Dear all,
    > > > >>>>>
    > > > >>>>> I am a heavy arcsight user and I have written quite
a few parsers
    > > > >> over
    > > > >>>>> time.
    > > > >>>>> I am new to contributing to open source projects however.
    > > > >>>>> @Ali, would you like to cooperate on development of
some parsers?
    > > > >>>>>
    > > > >>>>> Kind Regards,
    > > > >>>>> Mark de Rijk
    > > > >>>>>
    > > > >>>>>
    > > > >>>>>> On 7 Apr 2017, at 04:30, Ali Nazemian <alinazemian@gmail.com>
    > > > >> wrote:
    > > > >>>>>>
    > > > >>>>>> Hi all,
    > > > >>>>>>
    > > > >>>>>> We are going to develop some parsers and have
some contribution
    > to
    > > > >>> the
    > > > >>>>>> community as a start point. I was wondering what
the reason is
    > > > >> behind
    > > > >>>>>> choosing Grok statements for some of the implementations
and
    > Java
    > > > >>> regex
    > > > >>>>> for
    > > > >>>>>> other ones? Is there any policy for that? Probably
it would be
    > > > >> better
    > > > >>>> to
    > > > >>>>>> have the Java regex implementation due to performance
concerns.
    > > > >>>> However,
    > > > >>>>> I
    > > > >>>>>> am sure there is a reason that some of them have
been
    > implemented
    > > > >>> with
    > > > >>>>>> using Grok statements.
    > > > >>>>>>
    > > > >>>>>> Regards,
    > > > >>>>>> Ali
    > > > >>>>>
    > > > >>>>
    > > > >>>>
    > > > >>>>
    > > > >>>> --
    > > > >>>> A.Nazemian
    > > > >>>>
    > > > >>>
    > > > >>
    > > > >
    > > > >
    > > > >
    > > > > --
    > > > > A.Nazemian
    > > >
    > >
    > >
    > >
    > > --
    > > A.Nazemian
    > >
    >
    



Mime
View raw message