metron-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Matt Foley <ma...@apache.org>
Subject Re: [DISCUSS] SIDELOADING PARSERS: Packaging and Loading and Extensions [oh.my]
Date Fri, 10 Mar 2017 19:18:22 GMT
I like the approach.  I think Nar constitutes a production-quality existing solution meeting
highly similar needs to Metron’s.

Just a ‘btw’ regarding Joe’s input that I transmitted:
- Joe made clear that he was only giving his personal opinion, since of course no individual
can speak for the community.
- Joe also felt that if Metron succeeded in re-using the Nar system without having to change
it too much, that that would be a good supporting argument for later proposing that it become
a separate child project.
- Whereas if we or they tried to break it out as a separate project now, we would have to
do all the community-building work around it, as well as the technical work of adapting it
for a different environment from NiFi.
- So he recommended to copy and appropriate it for now.
- Which I also agree with.

Thanks,
--Matt

On 3/10/17, 7:42 AM, "Otto Fowler" <ottobackwards@gmail.com> wrote:

    As previously discussed here, I have been working on side loading of
    parsers.  The goals of this work are:
    * Make it possible of developers to create, maintain and deploy parsers
    outside of the Metron code tree and not have to fork
    * Create maven archetype support for developers of parsers
    * Introduce a parser ‘lifecycle’ to support multiple instances and
    configurations, states of being installed, under configuration, and deployed
    etc.
    
    I would like to have some discussion based on where I am after rebasing
    onto METRON-671 which revamps deployment to be totally ambari based.
    
    
    Packaging and Loading and Extensions
    
    I have mentioned previously, and we have discussed on list wanting to move
    away from uber’s for somethings to using custom class loaders ( from hdfs
    possibly ).
    We also want the REST api to work with 3rd parter parsers
    We would like to reduce the size of having so many ubers in the build
    We would benefit from tooling around this, maven building, archetypes etc
    We could benefit from explicitly required metadata and information
    We want a generic extension methodology
    We want to be able to upgrade parsers/extensions in some way
    
    
    I have also mentioned that this would look or work a lot like NiFi’s NAR
    system.
    
    Now I’m going to put it differently:
    
    I propose that we adapt and introduce the NAR system for Metron Extensions,
    starting with parsers, with that adoption
    extended to allow for VFS Classloading from hdfs as we are now doing with
    Stellar. And that this is done as a follow on to
    the base mvp side loading work.
    
    This provides a solution to the above issues, and would afford us a great
    amount of flexibility going forward.
    
    https://nifi.apache.org/docs/nifi-docs/html/developer-guide.html#nars
    https://cwiki.apache.org/confluence/display/NIFI/Maven+Projects+for+Extensions
    
    The functional concept would be:
     * the archetype and all the parser projects produce nars ( either
    including the configuration and patterns or splitting between runtime ( nar
    ) and static ( tar.gz )
    * these are not shaded, but have a ‘repo’ of dependencies for non-metron
    jars.  Metron jars are provided and loaded through the classloading.
    * possibly the adaption of the Service Provider api/pattern for parsers and
    discovery
    *  the nar repository/working directory structures would be implemented in
    /usr/metron/version/telemetry  ( although discussion on having multiple
    extension directories vs. one extension dir are welcome )
    * the storm process only references metron-parsers-common
    * the parser bolt uses the nar class loading system to load the parser
    * the rest api uses the nar class loading system to load the parser
    * etc etc
    * a new version of a parser is deployed as a nar, when the service
    restarts, the new nar is unpacked and replaces the old version in the
    working system ( we could change the restart requirement ……)
    
    The nar system gives us something that is:
    * production quality
    * small enough to grok and extend as opposed to some other solutions
    * comes with a highly accessible sister project
    * maven plugin tooling to build
    * reference archetype for packaging
    and other things
    
    There are a few ways we could approach using Nar:
    
    1. ‘fork’ and appropriate the components and ‘metronize’ them
    * the maven plugin
    * the nar-utils package
    2. Ask for and participate in an effort to pull NAR out into it’s own
    project,
    * make it more generic
    * usable by more than one project
    * goal to replace NiFi’s use of nar too
    3. Create our own generic version as a fork
    * use it in metron
    * submit to NiFi as a proposal
    
    MattF was nice enough to float this by Joe Witt, who is at Hortonworks and
    is the NiFi lead.
    He agrees with the idea of making nar usable for multiple projects, but
    does not see them
    having time to do it soon, and suggests we go with 1.
    
    I will openly admit to ignorance about apache subproject or other things
    that might come into play here.
    
    I think that 1 is the best option in the near term, and if we can socialize
    and see that there is a demand we can move forward with 2/3 down the road.
    
    ??
    Discuss
    Profit
    



Mime
View raw message