metron-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From James Sirota <jsir...@apache.org>
Subject Re: [DISCUSS] SIDELOADING PARSERS: Packaging and Loading and Extensions [oh.my]
Date Wed, 15 Mar 2017 03:49:15 GMT
Sounds like a great proposal. Im excited to see what you come up with.

james 

14.03.2017, 19:56, "Otto Fowler" <ottobackwards@gmail.com>:
> RE: questions -
>
> - The archetype will produce NARs so having it with the VFSClassloader
> doesn’t work, unless
> I do an archetype without nars, but that would be so short lived I wouldn’t
> bother.
> - My goal is to do it in such a way that I could propose / submit the
> changes made back to nifi ( worse case is work alike ).
> It is not out of the questions that between metron-nar ( or whatever I call
> it ) and nifi-nar, we come up with
> a 3rd thing that both switch to, that is generic and cross project. To
> that end, in the work I’m doing I am keeping dependencies
> to metron or nifi to zero.
>
> I am still doing some poc work. But the nar work ( that I will be putting
> into jira ) will be:
>
> * nar-utils ( the class loaders and expanders ) to a top level but not
> dependent metron library
> * refactor nar -utils to be independent of the nifi-properties and more
> generically configurable
> * replace the nifi file utils and file / uri class loaders with VFS
> FileSystemManager based implementation. ( this will work for them too, and
> I will be posting it back to them )
> * fork the nifi maven plugin and make it configurable as to output name
> extension and metadata names ( XXX-ID instead of Nar-ID ) - this too I
> will propose as a PR back
>
> I feel the right thing to do, when landing the parsers as separate, is not
> what I have done so far. Each parser should be a product of the archetype
> themselves.
> That is how I intend to proceed.
>
> On March 14, 2017 at 17:42:28, Kyle Richardson (kylerichardson2@gmail.com)
> wrote:
>
> I like the direction of using NAR. The key benefits I see (there are
> others, but I picked by favorites):
>
> 1. Move away from the uber jars in Storm. Granted, this should be possible
> with modifications from the new Stellar classloader. My hope is that this
> would allow us to stop shading the parser project packages.
> 2. Allow parser development to be somewhat independent of the core code
> base. I do think we'll have to address versioning and backwards
> compatibility.
> 3. "Drop in" extensions to Metron. Future expansion beyond parsers.
>
> Questions:
> - Is there a way to do 1 using the VFS Classloader and land the archetype
> as an MVP? If so, we could avoid shading as part of the archetype and maybe
> iterate from there on it. Do you think this would still be too much change
> to the archetype after putting it out there?
> - How would we adapt NAR from Nifi without frankensteining it? I'm all for
> code reuse but, ideally, we'd like Nifi and Metron's versions of NAR to end
> up converging in the end.
>
> On Fri, Mar 10, 2017 at 3:43 PM, Casey Stella <cestella@gmail.com> wrote:
>
>>  I would definitely agree that moving forward we should consider something
>>  like Nar for Stellar. I'm not seeing the need for parsers exactly.
>>
>>  I don't want to squash the forward thinking aspect here; we should be
>
> broad
>>  and think about the end, ideal state. I just want to make sure we think
>>  through something that we can iterate on as an initial state that still
>>  solves your problem, MVP style.
>>
>>  On Fri, Mar 10, 2017 at 3:39 PM, Otto Fowler <ottobackwards@gmail.com>
>>  wrote:
>>
>>  > "The Apache NiFi NAR ‘system’ allows for the packaging and loading of
>>  java
>>  > resources with classloader isolation.
>>  > Although technically it is the Service Provider api that makes the
>>  > ‘plugins’ part of the system, you can view them
>>  > together, and thus look at the NAR features as a system to create,
>>  > package, load, and execute plugins in a java system
>>  > while maintaining classloader isolation and dependency separation.
>>  >
>>  > While the NiFi problem case ( many plugins possibly executing in the
>
> same
>>  > vm ) is not universal, the functionality provided
>>  > by NAR is commonly needed, and is indeed functionality that I am
>>  currently
>>  > looking at implementing in the Apache Metron project.”
>>  >
>>  > This is how I put it to Joe.
>>  >
>>  > I think what you are proposing would work. I think what I have done up
>>  > until now will pretty much work. What I have been thinking about and
>>  > considering
>>  > is the difference between getting ‘something that works’, and maybe
>>  > something better.
>>  >
>>  > So if you look at nar there is the ‘packaging’ part, and the class
>>  loading
>>  > part.
>>  > We are already doing almost the same thing with the assembly of the
>>  > .tag.gz. The Nar is a next step to this which adds more metadata and
>
> the
>>  > dependency repo.
>>  > More of a refinement than a change.
>>  >
>>  > As far as class loading, the Nar is a more refined system for deploying
>>  > and consuming jars and dependencies, and setting up classloader
>>  instances.
>>  > It has more
>>  > functionality than we need at the moment in storm, but in other
>
> services
>>  > where multiple parsers or plugin types may need to be loaded, it would
>>  make
>>  > more sense.
>>  > Rest may be that case. Stellar may be that case too, if anyone ever
>>  > writes a stellar function with different dependencies than the
>
> platform.
>>  >
>>  >
>>  >
>>  > On March 10, 2017 at 14:32:00, Casey Stella (cestella@gmail.com) wrote:
>>  >
>>  > So, my question is whether we really need nar here. We have a
>>  classloading
>>  > mechanism that will allow us to deploy just the parser logic just added
>>  > into master for stellar, should we be considering another one?
>>  >
>>  > I would understand using nar if we needed to have multiple nars around
>>  > that
>>  > needed isolation from one another, but in the parser topology, we get
>>  that
>>  > isolation naturally. It seems to me that, at least for a MVP, we should
>>  > use the existing classloader that we just added. That being said, I
>
> might
>>  > be missing something, so let me know your thoughts.
>>  >
>>  > Casey
>>  >
>>  > On Fri, Mar 10, 2017 at 2:18 PM, Matt Foley <mattf@apache.org> wrote:
>>  >
>>  > > I like the approach. I think Nar constitutes a production-quality
>>  > > existing solution meeting highly similar needs to Metron’s.
>>  > >
>>  > > Just a ‘btw’ regarding Joe’s input that I transmitted:
>>  > > - Joe made clear that he was only giving his personal opinion, since
>
> of
>>  > > course no individual can speak for the community.
>>  > > - Joe also felt that if Metron succeeded in re-using the Nar system
>>  > > without having to change it too much, that that would be a good
>>  > supporting
>>  > > argument for later proposing that it become a separate child project.
>>  > > - Whereas if we or they tried to break it out as a separate project
>>  now,
>>  > > we would have to do all the community-building work around it, as
>
> well
>>  > as
>>  > > the technical work of adapting it for a different environment from
>>  NiFi.
>>  > > - So he recommended to copy and appropriate it for now.
>>  > > - Which I also agree with.
>>  > >
>>  > > Thanks,
>>  > > --Matt
>>  > >
>>  > > On 3/10/17, 7:42 AM, "Otto Fowler" <ottobackwards@gmail.com> wrote:
>>  > >
>>  > > As previously discussed here, I have been working on side loading of
>>  > > parsers. The goals of this work are:
>>  > > * Make it possible of developers to create, maintain and deploy
>
> parsers
>>  > > outside of the Metron code tree and not have to fork
>>  > > * Create maven archetype support for developers of parsers
>>  > > * Introduce a parser ‘lifecycle’ to support multiple instances and
>>  > > configurations, states of being installed, under configuration, and
>>  > > deployed
>>  > > etc.
>>  > >
>>  > > I would like to have some discussion based on where I am after
>
> rebasing
>>  > > onto METRON-671 which revamps deployment to be totally ambari based.
>>  > >
>>  > >
>>  > > Packaging and Loading and Extensions
>>  > >
>>  > > I have mentioned previously, and we have discussed on list wanting to
>>  > > move
>>  > > away from uber’s for somethings to using custom class loaders ( from
>>  > > hdfs
>>  > > possibly ).
>>  > > We also want the REST api to work with 3rd parter parsers
>>  > > We would like to reduce the size of having so many ubers in the build
>>  > > We would benefit from tooling around this, maven building, archetypes
>>  > > etc
>>  > > We could benefit from explicitly required metadata and information
>>  > > We want a generic extension methodology
>>  > > We want to be able to upgrade parsers/extensions in some way
>>  > >
>>  > >
>>  > > I have also mentioned that this would look or work a lot like NiFi’s
>>  > > NAR
>>  > > system.
>>  > >
>>  > > Now I’m going to put it differently:
>>  > >
>>  > > I propose that we adapt and introduce the NAR system for Metron
>>  > > Extensions,
>>  > > starting with parsers, with that adoption
>>  > > extended to allow for VFS Classloading from hdfs as we are now doing
>>  > > with
>>  > > Stellar. And that this is done as a follow on to
>>  > > the base mvp side loading work.
>>  > >
>>  > > This provides a solution to the above issues, and would afford us a
>>  > > great
>>  > > amount of flexibility going forward.
>>  > >
>>  > > https://nifi.apache.org/docs/nifi-docs/html/developer-guide.html#nars
>>  > > https://cwiki.apache.org/confluence/display/NIFI/Maven+
>>  > > Projects+for+Extensions
>>  > >
>>  > > The functional concept would be:
>>  > > * the archetype and all the parser projects produce nars ( either
>>  > > including the configuration and patterns or splitting between runtime
>>  > > ( nar
>>  > > ) and static ( tar.gz )
>>  > > * these are not shaded, but have a ‘repo’ of dependencies for
>>  > > non-metron
>>  > > jars. Metron jars are provided and loaded through the classloading.
>>  > > * possibly the adaption of the Service Provider api/pattern for
>>  > > parsers and
>>  > > discovery
>>  > > * the nar repository/working directory structures would be
>>  > > implemented in
>>  > > /usr/metron/version/telemetry ( although discussion on having
>
> multiple
>>  > > extension directories vs. one extension dir are welcome )
>>  > > * the storm process only references metron-parsers-common
>>  > > * the parser bolt uses the nar class loading system to load the
>
> parser
>>  > > * the rest api uses the nar class loading system to load the parser
>>  > > * etc etc
>>  > > * a new version of a parser is deployed as a nar, when the service
>>  > > restarts, the new nar is unpacked and replaces the old version in the
>>  > > working system ( we could change the restart requirement ……)
>>  > >
>>  > > The nar system gives us something that is:
>>  > > * production quality
>>  > > * small enough to grok and extend as opposed to some other solutions
>>  > > * comes with a highly accessible sister project
>>  > > * maven plugin tooling to build
>>  > > * reference archetype for packaging
>>  > > and other things
>>  > >
>>  > > There are a few ways we could approach using Nar:
>>  > >
>>  > > 1. ‘fork’ and appropriate the components and ‘metronize’ them
>>  > > * the maven plugin
>>  > > * the nar-utils package
>>  > > 2. Ask for and participate in an effort to pull NAR out into it’s own
>>  > > project,
>>  > > * make it more generic
>>  > > * usable by more than one project
>>  > > * goal to replace NiFi’s use of nar too
>>  > > 3. Create our own generic version as a fork
>>  > > * use it in metron
>>  > > * submit to NiFi as a proposal
>>  > >
>>  > > MattF was nice enough to float this by Joe Witt, who is at
>
> Hortonworks
>>  > > and
>>  > > is the NiFi lead.
>>  > > He agrees with the idea of making nar usable for multiple projects,
>
> but
>>  > > does not see them
>>  > > having time to do it soon, and suggests we go with 1.
>>  > >
>>  > > I will openly admit to ignorance about apache subproject or other
>>  > > things
>>  > > that might come into play here.
>>  > >
>>  > > I think that 1 is the best option in the near term, and if we can
>>  > > socialize
>>  > > and see that there is a demand we can move forward with 2/3 down the
>>  > > road.
>>  > >
>>  > > ??
>>  > > Discuss
>>  > > Profit
>>  > >
>>  > >
>>  > >
>>  > >
>>  >
>>  >

------------------- 
Thank you,

James Sirota
PPMC- Apache Metron (Incubating)
jsirota AT apache DOT org

Mime
View raw message