metron-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Kyle Richardson <kylerichards...@gmail.com>
Subject Re: [Discuss] SIDELOADING PARSERS: Parsers as components
Date Tue, 14 Mar 2017 13:57:09 GMT
Solid work, Otto. I'm excited to see us move in this direction. It's an
important step to making Metron a more user friendly platform.

I agree with Matt. I think a PR for this piece is needed sooner rather than
later so you don't continually have to rework it as we make more commits
into master.

One question/concern/future thought, is there anyway we can get away from
needing the RPMs for these extensible components? Ideally, I'd like to be
able to dump my parser package file (jar, nar, etc) into a directory and
have it automagically unpacked.

-Kyle

On Mon, Mar 13, 2017 at 10:10 AM, Otto Fowler <ottobackwards@gmail.com>
wrote:

> James,
> Can you clarify your concerns about backward compatibility?
>
>
>
> On March 12, 2017 at 23:48:17, James Sirota (jsirota@apache.org) wrote:
>
> As long as this doesn't break backwards compatibility I think I am ok with
> this approach. I think this is a great idea. We would probably need to
> follow up with an Ambari view that can allow you to list and
> deploy/upgrade/delete various parsers
>
> 10.03.2017, 15:16, "Casey Stella" <cestella@gmail.com>:
> > Ok, so, after some thought about this, I am in agreement over Nar. I do
> > want to make sure that on the roadmap we retrofit stellar to accept Nar
> > plugins and build an archetype for it. We should have a single strategy
> > for plugins. NOt saying it has to be part of the same PR, but it needs to
> > be associated and a follow-on task IMO.
> >
> > On Fri, Mar 10, 2017 at 4:06 PM, Otto Fowler <ottobackwards@gmail.com>
> > wrote:
> >
> >>  Also a Nar can depend on ‘one’ other nar, which is interesting
> >>
> >>  On March 10, 2017 at 16:02:18, Otto Fowler (ottobackwards@gmail.com)
> >>  wrote:
> >>
> >>  The isolation is just a ‘extra’ in the parser case.
> >>
> >>  The parts of Nar that *are* more pertinent:
> >>
> >>  * supporting a deployment artifact with just a jar, or a tar.gz with a
> jar
> >>  and jar dependencies in it
> >>  * taking a ‘package’ and deploying it for loading ( which will upgrade
> the
> >>  deployed part if it is newer )
> >>  * setting up the classloader hierarchy between the ‘system’ and
> provided
> >>  things, and the dependencies of the individual plugin
> >>
> >>  On March 10, 2017 at 15:56:08, Casey Stella (cestella@gmail.com)
> wrote:
> >>
> >>  Why would we need classpath isolation here in the case of the parser?
> >>
> >>  On Fri, Mar 10, 2017 at 3:55 PM, Otto Fowler <ottobackwards@gmail.com>
> >>  wrote:
> >>
> >>>  I *would* use the classloader part, extending it with VFS.
> >>>
> >>>  On March 10, 2017 at 15:53:05, Casey Stella (cestella@gmail.com)
> wrote:
> >>>
> >>>  I'm a bit worried about copying and pasting from the NiFi project
> their
> >>>  nar infrastructure. That seems..unclean to me and since we're not
> using
> >>>  the classloader part of nar for this, does it make more sense to just
> use
> >>>  jar?
> >>>
> >>>  On Fri, Mar 10, 2017 at 3:50 PM, Otto Fowler <ottobackwards@gmail.com
> >
> >>>  wrote:
> >>>
> >>>>  Compared to how much time vagrant up takes now, you won’t even notice
> it
> >>>>  ;)
> >>>>
> >>>>  That is definitely an option. I guess what I want to work out is if
> we
> >>>>  are going to want to
> >>>>  go to NAR, why not just go to NAR.
> >>>>
> >>>>  In the end, the customer for this - like Jon Zeolla, isn’t going
to
> care
> >>>>  about the intermediate step,
> >>>>  he wants the archetype that builds the ‘metron parser plugin’.
> >>>>
> >>>>  Which is why I hesitate to put out an archetype that is going to
> >>>>  obsolete so soon.
> >>>>
> >>>>  Does that make sense?
> >>>>
> >>>>  On March 10, 2017 at 14:50:55, Casey Stella (cestella@gmail.com)
> wrote:
> >>>>
> >>>>  I'm a little concerned about this increasing the size and length of
> the
> >>>>  build due to the repeated shading. Should we figure out a way to
> deploy
> >>>>  jars with provided dependencies on metron-parser-common as suggested
> in
> >>>>  the
> >>>>  previous JIRAs first?
> >>>>
> >>>>  On Fri, Mar 10, 2017 at 2:31 PM, Matt Foley <mfoley@hortonworks.com>
> >>>>  wrote:
> >>>>
> >>>>  > It sounds like:
> >>>>  > - This is a self-contained chunk of work, that can be tested,
> reviewed,
> >>>>  > and committed on its own, then the other ideas you propose can
> follow
> >>>>  it.
> >>>>  > - It crosses a lot of lines, and restructures a lot of code, so
> will
> >>>>  “rot”
> >>>>  > fairly quickly as other people make commits, so if possible you
> should
> >>>>  get
> >>>>  > a PR out there and we should work through it as soon as possible.
> >>>>  > Are those both true?
> >>>>  >
> >>>>  > How do other people feel about grouping a given sensor’s parser,
> >>>>  enricher,
> >>>>  > indexing logic all together? It seems to have multiple advantages
> are
> >>>>  > there also disadvantages?
> >>>>  >
> >>>>  > On 3/10/17, 6:31 AM, "Otto Fowler" <ottobackwards@gmail.com>
> wrote:
> >>>>  >
> >>>>  > As previously discussed here, I have been working on side loading
> of
> >>>>  > parsers. The goals of this work are:
> >>>>  > * Make it possible of developers to create, maintain and deploy
> parsers
> >>>>  > outside of the Metron code tree and not have to fork
> >>>>  > * Create maven archetype support for developers of parsers
> >>>>  > * Introduce a parser ‘lifecycle’ to support multiple instances
and
> >>>>  > configurations, states of being installed, under configuration,
and
> >>>>  > deployed
> >>>>  > etc.
> >>>>  >
> >>>>  > I would like to have some discussion based on where I am after
> rebasing
> >>>>  > onto METRON-671 which revamps deployment to be totally ambari
> based.
> >>>>  >
> >>>>  >
> >>>>  > Parsers as components:
> >>>>  >
> >>>>  > I have all the parsers broken out into individual
> packages/rpms/jars.
> >>>>  > What I have done is taken metron-parsers and broken it out to:
> >>>>  >
> >>>>  > * metron-parsers-common
> >>>>  > * This has all the base classes and interfaces, common testing
> >>>>  > components
> >>>>  > etc
> >>>>  > * metron-parser-base
> >>>>  > * This has the Grok, CSV, and JsonMap parsers and support
> >>>>  > * metron-parser-X
> >>>>  > * A module per parser type which we currently have in the system
> >>>>  > * Each parser has all the indexing, enrichment and parser
> >>>>  > configurations
> >>>>  > for that parser in its package
> >>>>  >
> >>>>  > I will go into packaging and deployment issues in another email.
> >>>>  >
> >>>>  > I have this all working:
> >>>>  > * the parsers are built
> >>>>  > * the parsers are tested
> >>>>  > * the parsers are integrated into the deployment build such that
> >>>>  > vagrant up
> >>>>  > just works as previously in full and quick dev
> >>>>  > * maven component of rpm docker
> >>>>  > * the metron.spec file
> >>>>  > * ambari installation
> >>>>  > * zookeeper configuration deployment
> >>>>  > * the ambari parser service code
> >>>>  > * the Rest interface works
> >>>>  > * see all installed parser configurations etc
> >>>>  >
> >>>>  >
> >>>>  > So this part of the work, is I think ready for a PR and review/next
> >>>>  > steps
> >>>>  > on it’s own.
> >>>>  >
> >>>>  > I think that it sets up the components and is a base for building
> out
> >>>>  > the
> >>>>  > rest of the functionality we want.
> >>>>  >
> >>>>  >
> >>>>  >
>
> -------------------
> Thank you,
>
> James Sirota
> PPMC- Apache Metron (Incubating)
> jsirota AT apache DOT org
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message