metron-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Otto Fowler <ottobackwa...@gmail.com>
Subject Re: [Discuss] SIDELOADING PARSERS: Parsers as components
Date Fri, 10 Mar 2017 20:43:18 GMT
Just to be clear, I didn’t group the logic of the indexing, enrichments and
parser, I grouped the configuration.  The idea is that a 3rd party developer
would like to produce a single product, that is self contained.

Yes those are both true, if we agree that this is a good way to go of
course.



On March 10, 2017 at 14:31:46, Matt Foley (mfoley@hortonworks.com) wrote:

It sounds like:
- This is a self-contained chunk of work, that can be tested, reviewed, and
committed on its own, then the other ideas you propose can follow it.
- It crosses a lot of lines, and restructures a lot of code, so will “rot”
fairly quickly as other people make commits, so if possible you should get
a PR out there and we should work through it as soon as possible.
Are those both true?

How do other people feel about grouping a given sensor’s parser, enricher,
indexing logic all together? It seems to have multiple advantages are there
also disadvantages?

On 3/10/17, 6:31 AM, "Otto Fowler" <ottobackwards@gmail.com> wrote:

As previously discussed here, I have been working on side loading of
parsers. The goals of this work are:
* Make it possible of developers to create, maintain and deploy parsers
outside of the Metron code tree and not have to fork
* Create maven archetype support for developers of parsers
* Introduce a parser ‘lifecycle’ to support multiple instances and
configurations, states of being installed, under configuration, and
deployed
etc.

I would like to have some discussion based on where I am after rebasing
onto METRON-671 which revamps deployment to be totally ambari based.


Parsers as components:

I have all the parsers broken out into individual packages/rpms/jars.
What I have done is taken metron-parsers and broken it out to:

* metron-parsers-common
* This has all the base classes and interfaces, common testing components
etc
* metron-parser-base
* This has the Grok, CSV, and JsonMap parsers and support
* metron-parser-X
* A module per parser type which we currently have in the system
* Each parser has all the indexing, enrichment and parser configurations
for that parser in its package

I will go into packaging and deployment issues in another email.

I have this all working:
* the parsers are built
* the parsers are tested
* the parsers are integrated into the deployment build such that vagrant up
just works as previously in full and quick dev
* maven component of rpm docker
* the metron.spec file
* ambari installation
* zookeeper configuration deployment
* the ambari parser service code
* the Rest interface works
* see all installed parser configurations etc


So this part of the work, is I think ready for a PR and review/next steps
on it’s own.

I think that it sets up the components and is a base for building out the
rest of the functionality we want.

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message