metron-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Otto Fowler <ottobackwa...@gmail.com>
Subject [DISCUSS] SIDELOADING PARSERS: Packaging and Deployment
Date Fri, 10 Mar 2017 14:53:09 GMT
As previously discussed here, I have been working on side loading of
parsers.  The goals of this work are:
* Make it possible of developers to create, maintain and deploy parsers
outside of the Metron code tree and not have to fork
* Create maven archetype support for developers of parsers
* Introduce a parser ‘lifecycle’ to support multiple instances and
configurations, states of being installed, under configuration, and deployed
etc.

I would like to have some discussion based on where I am after rebasing
onto METRON-671 which revamps deployment to be totally ambari based.


Packaging and Deployment

I have not change the packaging methodology that was already there, ie. All
the parsers are still shaded uber jars, and all are package into tar.gz to
include /lib /config /pattens

They are all explicitly called out in the copy resources portion of the
rpm-docker pom, and explicitly configured in the metron.spec for rpm
generation.

When deployed, they are deployed to a new directory - telemetry under the
metron home ( /usr/metron/0.3.1/telemetry ).
Each parser has it’s own directory, which gives it an isolated environment:

telemetry/asa
/config
/lib
/patterns

I could see adding a version here as well.  Also, this directory structure
could change, if not by review then by other follow on deployment options
due to ‘thinning'

All the scripts, ambari services have been changed to account for this, and
the start parser topology script is changed to find the right parser jar to
use for the -s option ( as opposed to only loading metron-parsers jar which
is the root of the issue ).

The packaging issues here:

* 10 or so new Uber jars increases the build size x2
* Travis needs to be changed from a container build to a vm build to get a
bigger space to work in

The rpm issues here:
* explicitly listing like items that *could* be iterated from a list is a
code smell to me.  With ansible I was able to define a list and use
with_items to get a nice clean, maintainable flow.  With rpm and maven
resources we have to have explicit entries.  My rpm and maven foo was not
good enough to sort this out, so I just bit the bullet and did it.  I think
we should explore copying using some kind of script or iteration in maven,
and possibly generating the metron spec from a template ( this too is
easier in ansible  ).

Options here:

1)  Accept this as mvp for the PR with improvements to packaging and
deployment as a follow on
2)  Delay the mvp and go for a possibly smaller - optimized deployment


Going for a smaller deployment, that is slimming the jars is something I’ll
talk about in another email :)

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message