metron-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Kyle Richardson <kylerichards...@gmail.com>
Subject Re: [Discuss] SIDELOADING PARSERS: Parsers as components
Date Tue, 14 Mar 2017 14:53:42 GMT
Gotcha. Makes sense.

Unrelated, but something I've had in the back of my head for a while... I
think we should try to loosely define what roles would be using Ambari vs
Metron UI vs CLI. For example, I'm thinking there may be a difference
between the cluster admin and the Metron admin in an organization. With
that, I think the idea of using the Metron UI to load parser packages makes
a lot of sense.

On Tue, Mar 14, 2017 at 10:39 AM, Otto Fowler <ottobackwards@gmail.com>
wrote:

> The RPM’s are only a requirement for installation by ambari.
>
> We could drop in Nars to the system location ( hdfs in the end ).
> I imagine us doing that through the UI.
>
> Nar’s *are* automatically unpacked at execution time.
>
> Nar’s are unpackaged into a ‘working’ area ( if required ), that working
> area is the class path that is
> loaded.
>
>
>
>
> On March 14, 2017 at 09:57:14, Kyle Richardson (kylerichardson2@gmail.com)
> wrote:
>
> Solid work, Otto. I'm excited to see us move in this direction. It's an
> important step to making Metron a more user friendly platform.
>
> I agree with Matt. I think a PR for this piece is needed sooner rather
> than
> later so you don't continually have to rework it as we make more commits
> into master.
>
> One question/concern/future thought, is there anyway we can get away from
> needing the RPMs for these extensible components? Ideally, I'd like to be
> able to dump my parser package file (jar, nar, etc) into a directory and
> have it automagically unpacked.
>
> -Kyle
>
> On Mon, Mar 13, 2017 at 10:10 AM, Otto Fowler <ottobackwards@gmail.com>
> wrote:
>
> > James,
> > Can you clarify your concerns about backward compatibility?
> >
> >
> >
> > On March 12, 2017 at 23:48:17, James Sirota (jsirota@apache.org) wrote:
> >
> > As long as this doesn't break backwards compatibility I think I am ok
> with
> > this approach. I think this is a great idea. We would probably need to
> > follow up with an Ambari view that can allow you to list and
> > deploy/upgrade/delete various parsers
> >
> > 10.03.2017, 15:16, "Casey Stella" <cestella@gmail.com>:
> > > Ok, so, after some thought about this, I am in agreement over Nar. I
> do
> > > want to make sure that on the roadmap we retrofit stellar to accept
> Nar
> > > plugins and build an archetype for it. We should have a single
> strategy
> > > for plugins. NOt saying it has to be part of the same PR, but it needs
> to
> > > be associated and a follow-on task IMO.
> > >
> > > On Fri, Mar 10, 2017 at 4:06 PM, Otto Fowler <ottobackwards@gmail.com>
>
> > > wrote:
> > >
> > >> Also a Nar can depend on ‘one’ other nar, which is interesting
> > >>
> > >> On March 10, 2017 at 16:02:18, Otto Fowler (ottobackwards@gmail.com)
> > >> wrote:
> > >>
> > >> The isolation is just a ‘extra’ in the parser case.
> > >>
> > >> The parts of Nar that *are* more pertinent:
> > >>
> > >> * supporting a deployment artifact with just a jar, or a tar.gz with
> a
> > jar
> > >> and jar dependencies in it
> > >> * taking a ‘package’ and deploying it for loading ( which will
> upgrade
> > the
> > >> deployed part if it is newer )
> > >> * setting up the classloader hierarchy between the ‘system’ and
> > provided
> > >> things, and the dependencies of the individual plugin
> > >>
> > >> On March 10, 2017 at 15:56:08, Casey Stella (cestella@gmail.com)
> > wrote:
> > >>
> > >> Why would we need classpath isolation here in the case of the parser?
> > >>
> > >> On Fri, Mar 10, 2017 at 3:55 PM, Otto Fowler <ottobackwards@gmail.com>
>
> > >> wrote:
> > >>
> > >>> I *would* use the classloader part, extending it with VFS.
> > >>>
> > >>> On March 10, 2017 at 15:53:05, Casey Stella (cestella@gmail.com)
> > wrote:
> > >>>
> > >>> I'm a bit worried about copying and pasting from the NiFi project
> > their
> > >>> nar infrastructure. That seems..unclean to me and since we're not
> > using
> > >>> the classloader part of nar for this, does it make more sense to
> just
> > use
> > >>> jar?
> > >>>
> > >>> On Fri, Mar 10, 2017 at 3:50 PM, Otto Fowler <
> ottobackwards@gmail.com
> > >
> > >>> wrote:
> > >>>
> > >>>> Compared to how much time vagrant up takes now, you won’t even
> notice
> > it
> > >>>> ;)
> > >>>>
> > >>>> That is definitely an option. I guess what I want to work out is
if
> > we
> > >>>> are going to want to
> > >>>> go to NAR, why not just go to NAR.
> > >>>>
> > >>>> In the end, the customer for this - like Jon Zeolla, isn’t going
to
> > care
> > >>>> about the intermediate step,
> > >>>> he wants the archetype that builds the ‘metron parser plugin’.
> > >>>>
> > >>>> Which is why I hesitate to put out an archetype that is going to
> > >>>> obsolete so soon.
> > >>>>
> > >>>> Does that make sense?
> > >>>>
> > >>>> On March 10, 2017 at 14:50:55, Casey Stella (cestella@gmail.com)
> > wrote:
> > >>>>
> > >>>> I'm a little concerned about this increasing the size and length
of
> > the
> > >>>> build due to the repeated shading. Should we figure out a way to
> > deploy
> > >>>> jars with provided dependencies on metron-parser-common as
> suggested
> > in
> > >>>> the
> > >>>> previous JIRAs first?
> > >>>>
> > >>>> On Fri, Mar 10, 2017 at 2:31 PM, Matt Foley <mfoley@hortonworks.com>
>
> > >>>> wrote:
> > >>>>
> > >>>> > It sounds like:
> > >>>> > - This is a self-contained chunk of work, that can be tested,
> > reviewed,
> > >>>> > and committed on its own, then the other ideas you propose
can
> > follow
> > >>>> it.
> > >>>> > - It crosses a lot of lines, and restructures a lot of code,
so
> > will
> > >>>> “rot”
> > >>>> > fairly quickly as other people make commits, so if possible
you
> > should
> > >>>> get
> > >>>> > a PR out there and we should work through it as soon as possible.
> > >>>> > Are those both true?
> > >>>> >
> > >>>> > How do other people feel about grouping a given sensor’s
parser,
> > >>>> enricher,
> > >>>> > indexing logic all together? It seems to have multiple advantages
> > are
> > >>>> > there also disadvantages?
> > >>>> >
> > >>>> > On 3/10/17, 6:31 AM, "Otto Fowler" <ottobackwards@gmail.com>
> > wrote:
> > >>>> >
> > >>>> > As previously discussed here, I have been working on side
loading
> > of
> > >>>> > parsers. The goals of this work are:
> > >>>> > * Make it possible of developers to create, maintain and deploy
> > parsers
> > >>>> > outside of the Metron code tree and not have to fork
> > >>>> > * Create maven archetype support for developers of parsers
> > >>>> > * Introduce a parser ‘lifecycle’ to support multiple instances
> and
> > >>>> > configurations, states of being installed, under configuration,
> and
> > >>>> > deployed
> > >>>> > etc.
> > >>>> >
> > >>>> > I would like to have some discussion based on where I am after
> > rebasing
> > >>>> > onto METRON-671 which revamps deployment to be totally ambari
> > based.
> > >>>> >
> > >>>> >
> > >>>> > Parsers as components:
> > >>>> >
> > >>>> > I have all the parsers broken out into individual
> > packages/rpms/jars.
> > >>>> > What I have done is taken metron-parsers and broken it out
to:
> > >>>> >
> > >>>> > * metron-parsers-common
> > >>>> > * This has all the base classes and interfaces, common testing
> > >>>> > components
> > >>>> > etc
> > >>>> > * metron-parser-base
> > >>>> > * This has the Grok, CSV, and JsonMap parsers and support
> > >>>> > * metron-parser-X
> > >>>> > * A module per parser type which we currently have in the
system
> > >>>> > * Each parser has all the indexing, enrichment and parser
> > >>>> > configurations
> > >>>> > for that parser in its package
> > >>>> >
> > >>>> > I will go into packaging and deployment issues in another
email.
> > >>>> >
> > >>>> > I have this all working:
> > >>>> > * the parsers are built
> > >>>> > * the parsers are tested
> > >>>> > * the parsers are integrated into the deployment build such
that
> > >>>> > vagrant up
> > >>>> > just works as previously in full and quick dev
> > >>>> > * maven component of rpm docker
> > >>>> > * the metron.spec file
> > >>>> > * ambari installation
> > >>>> > * zookeeper configuration deployment
> > >>>> > * the ambari parser service code
> > >>>> > * the Rest interface works
> > >>>> > * see all installed parser configurations etc
> > >>>> >
> > >>>> >
> > >>>> > So this part of the work, is I think ready for a PR and
> review/next
> > >>>> > steps
> > >>>> > on it’s own.
> > >>>> >
> > >>>> > I think that it sets up the components and is a base for building
> > out
> > >>>> > the
> > >>>> > rest of the functionality we want.
> > >>>> >
> > >>>> >
> > >>>> >
> >
> > -------------------
> > Thank you,
> >
> > James Sirota
> > PPMC- Apache Metron (Incubating)
> > jsirota AT apache DOT org
> >
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message