metron-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Casey Stella <ceste...@gmail.com>
Subject Re: [Discuss] SIDELOADING PARSERS: Parsers as components
Date Fri, 10 Mar 2017 20:54:21 GMT
Also, it's not clear that we're ever going to need the classloader bits
from nar for parsers since they are naturally isolated by storm topology.
I might be wrong there though; do you see a scenario?

On Fri, Mar 10, 2017 at 3:53 PM, Casey Stella <cestella@gmail.com> wrote:

> I'm a bit worried about copying and pasting from the NiFi project their
> nar infrastructure.  That seems..unclean to me and since we're not using
> the classloader part of nar for this, does it make more sense to just use
> jar?
>
> On Fri, Mar 10, 2017 at 3:50 PM, Otto Fowler <ottobackwards@gmail.com>
> wrote:
>
>> Compared to how much time vagrant up takes now, you won’t even notice it
>> ;)
>>
>> That is definitely an option.  I guess what I want to work out is if we
>> are going to want to
>> go to NAR, why not just go to NAR.
>>
>> In the end, the customer for this - like Jon Zeolla, isn’t going to care
>> about the intermediate step,
>> he wants the archetype that builds the ‘metron parser plugin’.
>>
>> Which is why I hesitate to put out an archetype that is going to obsolete
>> so soon.
>>
>> Does that make sense?
>>
>> On March 10, 2017 at 14:50:55, Casey Stella (cestella@gmail.com) wrote:
>>
>> I'm a little concerned about this increasing the size and length of the
>> build due to the repeated shading. Should we figure out a way to deploy
>> jars with provided dependencies on metron-parser-common as suggested in
>> the
>> previous JIRAs first?
>>
>> On Fri, Mar 10, 2017 at 2:31 PM, Matt Foley <mfoley@hortonworks.com>
>> wrote:
>>
>> > It sounds like:
>> > - This is a self-contained chunk of work, that can be tested, reviewed,
>> > and committed on its own, then the other ideas you propose can follow
>> it.
>> > - It crosses a lot of lines, and restructures a lot of code, so will
>> “rot”
>> > fairly quickly as other people make commits, so if possible you should
>> get
>> > a PR out there and we should work through it as soon as possible.
>> > Are those both true?
>> >
>> > How do other people feel about grouping a given sensor’s parser,
>> enricher,
>> > indexing logic all together? It seems to have multiple advantages are
>> > there also disadvantages?
>> >
>> > On 3/10/17, 6:31 AM, "Otto Fowler" <ottobackwards@gmail.com> wrote:
>> >
>> > As previously discussed here, I have been working on side loading of
>> > parsers. The goals of this work are:
>> > * Make it possible of developers to create, maintain and deploy parsers
>> > outside of the Metron code tree and not have to fork
>> > * Create maven archetype support for developers of parsers
>> > * Introduce a parser ‘lifecycle’ to support multiple instances and
>> > configurations, states of being installed, under configuration, and
>> > deployed
>> > etc.
>> >
>> > I would like to have some discussion based on where I am after rebasing
>> > onto METRON-671 which revamps deployment to be totally ambari based.
>> >
>> >
>> > Parsers as components:
>> >
>> > I have all the parsers broken out into individual packages/rpms/jars.
>> > What I have done is taken metron-parsers and broken it out to:
>> >
>> > * metron-parsers-common
>> > * This has all the base classes and interfaces, common testing
>> > components
>> > etc
>> > * metron-parser-base
>> > * This has the Grok, CSV, and JsonMap parsers and support
>> > * metron-parser-X
>> > * A module per parser type which we currently have in the system
>> > * Each parser has all the indexing, enrichment and parser
>> > configurations
>> > for that parser in its package
>> >
>> > I will go into packaging and deployment issues in another email.
>> >
>> > I have this all working:
>> > * the parsers are built
>> > * the parsers are tested
>> > * the parsers are integrated into the deployment build such that
>> > vagrant up
>> > just works as previously in full and quick dev
>> > * maven component of rpm docker
>> > * the metron.spec file
>> > * ambari installation
>> > * zookeeper configuration deployment
>> > * the ambari parser service code
>> > * the Rest interface works
>> > * see all installed parser configurations etc
>> >
>> >
>> > So this part of the work, is I think ready for a PR and review/next
>> > steps
>> > on it’s own.
>> >
>> > I think that it sets up the components and is a base for building out
>> > the
>> > rest of the functionality we want.
>> >
>> >
>> >
>>
>>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message