metron-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Casey Stella <ceste...@gmail.com>
Subject Re: [Discuss] SIDELOADING PARSERS: Parsers as components
Date Fri, 10 Mar 2017 22:16:32 GMT
Ok, so, after some thought about this, I am in agreement over Nar.  I do
want to make sure that on the roadmap we retrofit stellar to accept Nar
plugins and build an archetype for it.  We should have a single strategy
for plugins.  NOt saying it has to be part of the same PR, but it needs to
be associated and a follow-on task IMO.

On Fri, Mar 10, 2017 at 4:06 PM, Otto Fowler <ottobackwards@gmail.com>
wrote:

> Also a Nar can depend on ‘one’ other nar, which is interesting
>
>
> On March 10, 2017 at 16:02:18, Otto Fowler (ottobackwards@gmail.com)
> wrote:
>
> The isolation is just a ‘extra’ in the parser case.
>
> The parts of Nar that *are* more pertinent:
>
> * supporting a deployment artifact with just a jar, or a tar.gz with a jar
> and jar dependencies in it
> * taking a ‘package’ and deploying it for loading ( which will upgrade the
> deployed part if it is newer )
> * setting up the classloader hierarchy between the ‘system’ and provided
> things, and the dependencies of the individual plugin
>
>
>
> On March 10, 2017 at 15:56:08, Casey Stella (cestella@gmail.com) wrote:
>
> Why would we need classpath isolation here in the case of the parser?
>
> On Fri, Mar 10, 2017 at 3:55 PM, Otto Fowler <ottobackwards@gmail.com>
> wrote:
>
>> I *would* use the classloader part, extending it with VFS.
>>
>>
>>
>> On March 10, 2017 at 15:53:05, Casey Stella (cestella@gmail.com) wrote:
>>
>> I'm a bit worried about copying and pasting from the NiFi project their
>> nar infrastructure.  That seems..unclean to me and since we're not using
>> the classloader part of nar for this, does it make more sense to just use
>> jar?
>>
>> On Fri, Mar 10, 2017 at 3:50 PM, Otto Fowler <ottobackwards@gmail.com>
>> wrote:
>>
>>> Compared to how much time vagrant up takes now, you won’t even notice it
>>> ;)
>>>
>>> That is definitely an option.  I guess what I want to work out is if we
>>> are going to want to
>>> go to NAR, why not just go to NAR.
>>>
>>> In the end, the customer for this - like Jon Zeolla, isn’t going to care
>>> about the intermediate step,
>>> he wants the archetype that builds the ‘metron parser plugin’.
>>>
>>> Which is why I hesitate to put out an archetype that is going to
>>> obsolete so soon.
>>>
>>> Does that make sense?
>>>
>>> On March 10, 2017 at 14:50:55, Casey Stella (cestella@gmail.com) wrote:
>>>
>>> I'm a little concerned about this increasing the size and length of the
>>> build due to the repeated shading. Should we figure out a way to deploy
>>> jars with provided dependencies on metron-parser-common as suggested in
>>> the
>>> previous JIRAs first?
>>>
>>> On Fri, Mar 10, 2017 at 2:31 PM, Matt Foley <mfoley@hortonworks.com>
>>> wrote:
>>>
>>> > It sounds like:
>>> > - This is a self-contained chunk of work, that can be tested, reviewed,
>>> > and committed on its own, then the other ideas you propose can follow
>>> it.
>>> > - It crosses a lot of lines, and restructures a lot of code, so will
>>> “rot”
>>> > fairly quickly as other people make commits, so if possible you should
>>> get
>>> > a PR out there and we should work through it as soon as possible.
>>> > Are those both true?
>>> >
>>> > How do other people feel about grouping a given sensor’s parser,
>>> enricher,
>>> > indexing logic all together? It seems to have multiple advantages are
>>> > there also disadvantages?
>>> >
>>> > On 3/10/17, 6:31 AM, "Otto Fowler" <ottobackwards@gmail.com> wrote:
>>> >
>>> > As previously discussed here, I have been working on side loading of
>>> > parsers. The goals of this work are:
>>> > * Make it possible of developers to create, maintain and deploy parsers
>>> > outside of the Metron code tree and not have to fork
>>> > * Create maven archetype support for developers of parsers
>>> > * Introduce a parser ‘lifecycle’ to support multiple instances and
>>> > configurations, states of being installed, under configuration, and
>>> > deployed
>>> > etc.
>>> >
>>> > I would like to have some discussion based on where I am after rebasing
>>> > onto METRON-671 which revamps deployment to be totally ambari based.
>>> >
>>> >
>>> > Parsers as components:
>>> >
>>> > I have all the parsers broken out into individual packages/rpms/jars.
>>> > What I have done is taken metron-parsers and broken it out to:
>>> >
>>> > * metron-parsers-common
>>> > * This has all the base classes and interfaces, common testing
>>> > components
>>> > etc
>>> > * metron-parser-base
>>> > * This has the Grok, CSV, and JsonMap parsers and support
>>> > * metron-parser-X
>>> > * A module per parser type which we currently have in the system
>>> > * Each parser has all the indexing, enrichment and parser
>>> > configurations
>>> > for that parser in its package
>>> >
>>> > I will go into packaging and deployment issues in another email.
>>> >
>>> > I have this all working:
>>> > * the parsers are built
>>> > * the parsers are tested
>>> > * the parsers are integrated into the deployment build such that
>>> > vagrant up
>>> > just works as previously in full and quick dev
>>> > * maven component of rpm docker
>>> > * the metron.spec file
>>> > * ambari installation
>>> > * zookeeper configuration deployment
>>> > * the ambari parser service code
>>> > * the Rest interface works
>>> > * see all installed parser configurations etc
>>> >
>>> >
>>> > So this part of the work, is I think ready for a PR and review/next
>>> > steps
>>> > on it’s own.
>>> >
>>> > I think that it sets up the components and is a base for building out
>>> > the
>>> > rest of the functionality we want.
>>> >
>>> >
>>> >
>>>
>>>
>>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message