metron-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Matt Foley <mfo...@hortonworks.com>
Subject Re: [Discuss] SIDELOADING PARSERS: Parsers as components
Date Fri, 10 Mar 2017 19:31:35 GMT
It sounds like:
- This is a self-contained chunk of work, that can be tested, reviewed, and committed on its
own, then the other ideas you propose can follow it.
- It crosses a lot of lines, and restructures a lot of code, so will “rot” fairly quickly
as other people make commits, so if possible you should get a PR out there and we should work
through it as soon as possible.
Are those both true?

How do other people feel about grouping a given sensor’s parser, enricher, indexing logic
all together?  It seems to have multiple advantages are there also disadvantages?

On 3/10/17, 6:31 AM, "Otto Fowler" <ottobackwards@gmail.com> wrote:

    As previously discussed here, I have been working on side loading of
    parsers.  The goals of this work are:
    * Make it possible of developers to create, maintain and deploy parsers
    outside of the Metron code tree and not have to fork
    * Create maven archetype support for developers of parsers
    * Introduce a parser ‘lifecycle’ to support multiple instances and
    configurations, states of being installed, under configuration, and deployed
    etc.
    
    I would like to have some discussion based on where I am after rebasing
    onto METRON-671 which revamps deployment to be totally ambari based.
    
    
    Parsers as components:
    
    I have all the parsers broken out into individual packages/rpms/jars.
    What I have done is taken metron-parsers and broken it out to:
    
    * metron-parsers-common
    * This has all the base classes and interfaces, common testing components
    etc
    * metron-parser-base
    * This has the Grok, CSV, and JsonMap parsers and support
    * metron-parser-X
    * A module per parser type which we currently have in the system
    * Each parser has all the indexing, enrichment and parser configurations
    for that parser in its package
    
    I will go into packaging and deployment issues in another email.
    
    I have this all working:
    * the parsers are built
    * the parsers are tested
    * the parsers are integrated into the deployment build such that vagrant up
    just works as previously in full and quick dev
    * maven component of rpm docker
      * the metron.spec file
    * ambari installation
    * zookeeper configuration deployment
    * the ambari parser service code
    * the Rest interface works
    * see all installed parser configurations etc
    
    
    So this part of the work, is I think ready for a PR and review/next steps
    on it’s own.
    
    I think that it sets up the components and is a base for building out the
    rest of the functionality we want.
    

Mime
View raw message