metron-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Otto Fowler <ottobackwa...@gmail.com>
Subject Re: [Discuss] SIDELOADING PARSERS: Parsers as components
Date Fri, 10 Mar 2017 20:50:45 GMT
Compared to how much time vagrant up takes now, you won’t even notice it ;)

That is definitely an option.  I guess what I want to work out is if we are
going to want to
go to NAR, why not just go to NAR.

In the end, the customer for this - like Jon Zeolla, isn’t going to care
about the intermediate step,
he wants the archetype that builds the ‘metron parser plugin’.

Which is why I hesitate to put out an archetype that is going to obsolete
so soon.

Does that make sense?

On March 10, 2017 at 14:50:55, Casey Stella (cestella@gmail.com) wrote:

I'm a little concerned about this increasing the size and length of the
build due to the repeated shading. Should we figure out a way to deploy
jars with provided dependencies on metron-parser-common as suggested in the
previous JIRAs first?

On Fri, Mar 10, 2017 at 2:31 PM, Matt Foley <mfoley@hortonworks.com> wrote:

> It sounds like:
> - This is a self-contained chunk of work, that can be tested, reviewed,
> and committed on its own, then the other ideas you propose can follow it.
> - It crosses a lot of lines, and restructures a lot of code, so will
“rot”
> fairly quickly as other people make commits, so if possible you should
get
> a PR out there and we should work through it as soon as possible.
> Are those both true?
>
> How do other people feel about grouping a given sensor’s parser,
enricher,
> indexing logic all together? It seems to have multiple advantages are
> there also disadvantages?
>
> On 3/10/17, 6:31 AM, "Otto Fowler" <ottobackwards@gmail.com> wrote:
>
> As previously discussed here, I have been working on side loading of
> parsers. The goals of this work are:
> * Make it possible of developers to create, maintain and deploy parsers
> outside of the Metron code tree and not have to fork
> * Create maven archetype support for developers of parsers
> * Introduce a parser ‘lifecycle’ to support multiple instances and
> configurations, states of being installed, under configuration, and
> deployed
> etc.
>
> I would like to have some discussion based on where I am after rebasing
> onto METRON-671 which revamps deployment to be totally ambari based.
>
>
> Parsers as components:
>
> I have all the parsers broken out into individual packages/rpms/jars.
> What I have done is taken metron-parsers and broken it out to:
>
> * metron-parsers-common
> * This has all the base classes and interfaces, common testing
> components
> etc
> * metron-parser-base
> * This has the Grok, CSV, and JsonMap parsers and support
> * metron-parser-X
> * A module per parser type which we currently have in the system
> * Each parser has all the indexing, enrichment and parser
> configurations
> for that parser in its package
>
> I will go into packaging and deployment issues in another email.
>
> I have this all working:
> * the parsers are built
> * the parsers are tested
> * the parsers are integrated into the deployment build such that
> vagrant up
> just works as previously in full and quick dev
> * maven component of rpm docker
> * the metron.spec file
> * ambari installation
> * zookeeper configuration deployment
> * the ambari parser service code
> * the Rest interface works
> * see all installed parser configurations etc
>
>
> So this part of the work, is I think ready for a PR and review/next
> steps
> on it’s own.
>
> I think that it sets up the components and is a base for building out
> the
> rest of the functionality we want.
>
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message