metron-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From ottobackwards <...@git.apache.org>
Subject [GitHub] metron issue #530: METRON-777 Metron Extension System and Parser Extensions
Date Mon, 31 Jul 2017 03:46:03 GMT
Github user ottobackwards commented on the issue:

    https://github.com/apache/metron/pull/530
  
    "Based on feedback from reviews and discussion with mattf, this commit…
    … represent a major refactoring for the bundle functionality.
    
    While trying to resolve the ugliness bought about by working around VFS's read only support
for HDFS, I have come up with a different approach.
    
    While this approach makes bundles-lib significantly different than nar-utils, it makes
it much more simple, and may arguably be an approach worth
    concidering.
    
    The original bundles-lib maintained the nar system's method of unpacking the bundle into
a 'working' directory, and then using having the classloader
    load from that directory structure.  Trying to avoid too far a departure ( at the time
) from nifi, bundle-lib maintained the 'create folders and files as nec'
    scheme found in nifi.  Since nifi is always working with local disk and File objects,
this is ok. But with VFS and HDFS this did not work.  The work around
    was to create a poop static 'class cluster' type step with multiple implemenations of
FileUtils ( VFS and HDFS ).
    
    The new approach, which is a more willing departure from the nifi approach, is to avoid
having to ever do writes ( file or folder creation ) from bundles-lib at all.
    This allows for a simplier, single implementation of the file utils and the removal of
the anti-pattern.  It also removes the need for a 'working' area.
    
    The new approach is to better utilize the capabilities of VFS such that we never have
to unpack the bundle at all.  VFS can load JAR files as Filesystems. I have created
    a new Classloader ( based on the VFS classloader implemenation from the VFS project )
that is bundle aware.  It can load the bundle and it's dependencies as resources
    and make them available.  This is what removes the requirement to unpack the bundles into
a working directory.
    
    As such, the bundle-lib has been refactored to move from the 'working directory' to only
needing to work with the bundle in the lib.  I have also removed the doc directory support
    since we are not using it.
    
    We no longer need multiple file utils, the hdfs working directory, and other things.
    
    Thanks to mattf for the time and discussion."


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

Mime
View raw message