metron-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Otto Fowler <ottobackwa...@gmail.com>
Subject Re: [Discuss] SIDELOADING PARSERS: Parsers as components
Date Mon, 13 Mar 2017 14:10:00 GMT
James,
Can you clarify your concerns about backward compatibility?



On March 12, 2017 at 23:48:17, James Sirota (jsirota@apache.org) wrote:

As long as this doesn't break backwards compatibility I think I am ok with
this approach. I think this is a great idea. We would probably need to
follow up with an Ambari view that can allow you to list and
deploy/upgrade/delete various parsers

10.03.2017, 15:16, "Casey Stella" <cestella@gmail.com>:
> Ok, so, after some thought about this, I am in agreement over Nar. I do
> want to make sure that on the roadmap we retrofit stellar to accept Nar
> plugins and build an archetype for it. We should have a single strategy
> for plugins. NOt saying it has to be part of the same PR, but it needs to
> be associated and a follow-on task IMO.
>
> On Fri, Mar 10, 2017 at 4:06 PM, Otto Fowler <ottobackwards@gmail.com>
> wrote:
>
>>  Also a Nar can depend on ‘one’ other nar, which is interesting
>>
>>  On March 10, 2017 at 16:02:18, Otto Fowler (ottobackwards@gmail.com)
>>  wrote:
>>
>>  The isolation is just a ‘extra’ in the parser case.
>>
>>  The parts of Nar that *are* more pertinent:
>>
>>  * supporting a deployment artifact with just a jar, or a tar.gz with a
jar
>>  and jar dependencies in it
>>  * taking a ‘package’ and deploying it for loading ( which will upgrade
the
>>  deployed part if it is newer )
>>  * setting up the classloader hierarchy between the ‘system’ and
provided
>>  things, and the dependencies of the individual plugin
>>
>>  On March 10, 2017 at 15:56:08, Casey Stella (cestella@gmail.com) wrote:
>>
>>  Why would we need classpath isolation here in the case of the parser?
>>
>>  On Fri, Mar 10, 2017 at 3:55 PM, Otto Fowler <ottobackwards@gmail.com>
>>  wrote:
>>
>>>  I *would* use the classloader part, extending it with VFS.
>>>
>>>  On March 10, 2017 at 15:53:05, Casey Stella (cestella@gmail.com)
wrote:
>>>
>>>  I'm a bit worried about copying and pasting from the NiFi project
their
>>>  nar infrastructure. That seems..unclean to me and since we're not
using
>>>  the classloader part of nar for this, does it make more sense to just
use
>>>  jar?
>>>
>>>  On Fri, Mar 10, 2017 at 3:50 PM, Otto Fowler <ottobackwards@gmail.com>
>>>  wrote:
>>>
>>>>  Compared to how much time vagrant up takes now, you won’t even notice
it
>>>>  ;)
>>>>
>>>>  That is definitely an option. I guess what I want to work out is if
we
>>>>  are going to want to
>>>>  go to NAR, why not just go to NAR.
>>>>
>>>>  In the end, the customer for this - like Jon Zeolla, isn’t going to
care
>>>>  about the intermediate step,
>>>>  he wants the archetype that builds the ‘metron parser plugin’.
>>>>
>>>>  Which is why I hesitate to put out an archetype that is going to
>>>>  obsolete so soon.
>>>>
>>>>  Does that make sense?
>>>>
>>>>  On March 10, 2017 at 14:50:55, Casey Stella (cestella@gmail.com)
wrote:
>>>>
>>>>  I'm a little concerned about this increasing the size and length of
the
>>>>  build due to the repeated shading. Should we figure out a way to
deploy
>>>>  jars with provided dependencies on metron-parser-common as suggested
in
>>>>  the
>>>>  previous JIRAs first?
>>>>
>>>>  On Fri, Mar 10, 2017 at 2:31 PM, Matt Foley <mfoley@hortonworks.com>
>>>>  wrote:
>>>>
>>>>  > It sounds like:
>>>>  > - This is a self-contained chunk of work, that can be tested,
reviewed,
>>>>  > and committed on its own, then the other ideas you propose can
follow
>>>>  it.
>>>>  > - It crosses a lot of lines, and restructures a lot of code, so
will
>>>>  “rot”
>>>>  > fairly quickly as other people make commits, so if possible you
should
>>>>  get
>>>>  > a PR out there and we should work through it as soon as possible.
>>>>  > Are those both true?
>>>>  >
>>>>  > How do other people feel about grouping a given sensor’s parser,
>>>>  enricher,
>>>>  > indexing logic all together? It seems to have multiple advantages
are
>>>>  > there also disadvantages?
>>>>  >
>>>>  > On 3/10/17, 6:31 AM, "Otto Fowler" <ottobackwards@gmail.com>
wrote:
>>>>  >
>>>>  > As previously discussed here, I have been working on side loading
of
>>>>  > parsers. The goals of this work are:
>>>>  > * Make it possible of developers to create, maintain and deploy
parsers
>>>>  > outside of the Metron code tree and not have to fork
>>>>  > * Create maven archetype support for developers of parsers
>>>>  > * Introduce a parser ‘lifecycle’ to support multiple instances
and
>>>>  > configurations, states of being installed, under configuration, and
>>>>  > deployed
>>>>  > etc.
>>>>  >
>>>>  > I would like to have some discussion based on where I am after
rebasing
>>>>  > onto METRON-671 which revamps deployment to be totally ambari
based.
>>>>  >
>>>>  >
>>>>  > Parsers as components:
>>>>  >
>>>>  > I have all the parsers broken out into individual
packages/rpms/jars.
>>>>  > What I have done is taken metron-parsers and broken it out to:
>>>>  >
>>>>  > * metron-parsers-common
>>>>  > * This has all the base classes and interfaces, common testing
>>>>  > components
>>>>  > etc
>>>>  > * metron-parser-base
>>>>  > * This has the Grok, CSV, and JsonMap parsers and support
>>>>  > * metron-parser-X
>>>>  > * A module per parser type which we currently have in the system
>>>>  > * Each parser has all the indexing, enrichment and parser
>>>>  > configurations
>>>>  > for that parser in its package
>>>>  >
>>>>  > I will go into packaging and deployment issues in another email.
>>>>  >
>>>>  > I have this all working:
>>>>  > * the parsers are built
>>>>  > * the parsers are tested
>>>>  > * the parsers are integrated into the deployment build such that
>>>>  > vagrant up
>>>>  > just works as previously in full and quick dev
>>>>  > * maven component of rpm docker
>>>>  > * the metron.spec file
>>>>  > * ambari installation
>>>>  > * zookeeper configuration deployment
>>>>  > * the ambari parser service code
>>>>  > * the Rest interface works
>>>>  > * see all installed parser configurations etc
>>>>  >
>>>>  >
>>>>  > So this part of the work, is I think ready for a PR and review/next
>>>>  > steps
>>>>  > on it’s own.
>>>>  >
>>>>  > I think that it sets up the components and is a base for building
out
>>>>  > the
>>>>  > rest of the functionality we want.
>>>>  >
>>>>  >
>>>>  >

-------------------
Thank you,

James Sirota
PPMC- Apache Metron (Incubating)
jsirota AT apache DOT org

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message