metron-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Casey Stella <ceste...@gmail.com>
Subject Re: [DISCUSS] SIDELOADING PARSERS: Packaging and Deployment
Date Fri, 10 Mar 2017 17:56:15 GMT
I really don't like the 2x build size.  I think we can at this point do
something similar to side-loading of stellar functions to remove that
concern. This should be easy now that that's in master.

What I'd like to see as a MVP is:

   - the maven archetype to have a "provided" dependency to metron-parsers
   - Create a JIRA to
      - Add a "parser.paths" field to global config
      - modify the ParserBolt to use the VFSClassloader to instantiate the
      parser

I'd consider that phase 1 as it solves the question of "how do I easily
create parsers without forking Metron" and it doesn't require shaded jars.

>From there, I wouldn't be opposed to splitting the existing parsers into
separate projects, but I'd consider that a phase 2 activity.  We can trim
down metron-parsers at our leisure.

Thoughts?

On Fri, Mar 10, 2017 at 9:53 AM, Otto Fowler <ottobackwards@gmail.com>
wrote:

> As previously discussed here, I have been working on side loading of
> parsers.  The goals of this work are:
> * Make it possible of developers to create, maintain and deploy parsers
> outside of the Metron code tree and not have to fork
> * Create maven archetype support for developers of parsers
> * Introduce a parser ‘lifecycle’ to support multiple instances and
> configurations, states of being installed, under configuration, and
> deployed
> etc.
>
> I would like to have some discussion based on where I am after rebasing
> onto METRON-671 which revamps deployment to be totally ambari based.
>
>
> Packaging and Deployment
>
> I have not change the packaging methodology that was already there, ie. All
> the parsers are still shaded uber jars, and all are package into tar.gz to
> include /lib /config /pattens
>
> They are all explicitly called out in the copy resources portion of the
> rpm-docker pom, and explicitly configured in the metron.spec for rpm
> generation.
>
> When deployed, they are deployed to a new directory - telemetry under the
> metron home ( /usr/metron/0.3.1/telemetry ).
> Each parser has it’s own directory, which gives it an isolated environment:
>
> telemetry/asa
> /config
> /lib
> /patterns
>
> I could see adding a version here as well.  Also, this directory structure
> could change, if not by review then by other follow on deployment options
> due to ‘thinning'
>
> All the scripts, ambari services have been changed to account for this, and
> the start parser topology script is changed to find the right parser jar to
> use for the -s option ( as opposed to only loading metron-parsers jar which
> is the root of the issue ).
>
> The packaging issues here:
>
> * 10 or so new Uber jars increases the build size x2
> * Travis needs to be changed from a container build to a vm build to get a
> bigger space to work in
>
> The rpm issues here:
> * explicitly listing like items that *could* be iterated from a list is a
> code smell to me.  With ansible I was able to define a list and use
> with_items to get a nice clean, maintainable flow.  With rpm and maven
> resources we have to have explicit entries.  My rpm and maven foo was not
> good enough to sort this out, so I just bit the bullet and did it.  I think
> we should explore copying using some kind of script or iteration in maven,
> and possibly generating the metron spec from a template ( this too is
> easier in ansible  ).
>
> Options here:
>
> 1)  Accept this as mvp for the PR with improvements to packaging and
> deployment as a follow on
> 2)  Delay the mvp and go for a possibly smaller - optimized deployment
>
>
> Going for a smaller deployment, that is slimming the jars is something I’ll
> talk about in another email :)
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message