metron-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Zeolla@GMail.com" <zeo...@gmail.com>
Subject Re: [DISCUSS] Ambari Integration
Date Thu, 15 Sep 2016 22:50:09 GMT
First of all - very much looking forward to this approach.  I'm not very
familiar with management packs, but I did read some of the documentation in
the link you sent.

Not sure if this is already included in a "minimum viable product," but at
some point I think there needs to be a method of specifying proxies and/or
internal package repos.  I recently did a Metron 0.2.0 install behind a
proxy (hence METRON-409 <https://issues.apache.org/jira/browse/METRON-409>)
and it look me a semi-lengthy amount of time to (1) find all of the
destinations I needed to request openings for in the proxy, and (2) modify
the ambari scripts to appropriately use my proxies in the correct way.

I also have a bit of a concern with upgrades and customizations in general
(Not just how it would work with mpacks).  I have not done any of this to
date, but I have rebuilt and redeployed a couple of times and I needed to
modify some of the metron code itself before build/deploy (because of my
concern with it getting overwritten on upgrade if I just did it directly on
the cluster).  I would like to see a method of putting in install-specific
files that modify or overwrite parts of the core metron stack, like changes
to znodes, parsers, etc.

Regarding not managing sensors with Ambari, I agree.  I run a large bro
cluster and it is maintained via Puppet and various other mechanisms - no
need for Ambari to bleed over in my case.

Thanks for the great work.

Jon

On Thu, Sep 15, 2016 at 5:10 PM Casey Stella <cestella@gmail.com> wrote:

> Hi Everyone,
>
> I wanted to solicit some discussion around a feature that is fast
> approaching.  A major pain point in using Metron is installation.  Thus far
> our only approach to installation has been driven by the developer's needs
> to construct a virtual environment to test out changes, which lead us to
> either an ansible installation or a manual installation.
>
> Because we want to make sure that the installation of Metron is as easy as
> possible, we have had some great contributions of an additional approach,
> installation via Apache Ambari directly.  Our ansible scripts currently
> rely on Ambari blueprints to set up Hadoop on the cluster that it is
> deploying on, so it is not a new dependency, but we're working toward a
> full Ambari management pack
> <https://cwiki.apache.org/confluence/display/AMBARI/Management+Packs>
> that will lay down the relevant topologies (parser, enrichment, indexing),
> configs, bits and their infrastructural dependencies (ES and mysql) and
> allow the topologies to be started and stopped as minimum viable product.
>
> The beginnings of this have started with:
>
>    - Ambari Service Definitions for the Parser topologies
>    <https://github.com/apache/incubator-metron/pull/218>
>    - Ambari Service Definition for the Indexing Topology
>    <https://github.com/apache/incubator-metron/pull/222>
>    - Ambari Service Definition for Elasticsearch
>    <https://github.com/apache/incubator-metron/pull/223>
>
> There will be more to come in the near-term to realize that vision, but we
> wanted to get some reactions.  Past minimum viable product, what do you
> guys think we should have and how should it look?
>
> Currently we are treating the domain of the ambari installation as from
> kafka to the indexes, which leaves the sensors unmanaged via ambari.  Is
> that a good decision?
>
> Are there other pain points that you have had around installation that
> you'd like to see addressed?
>
> The purpose of this discussion thread is to let you guys know that we will
> soon have a new way to install metron, but also to understand what the
> future requirements are so we, as a community, can address them.
>
> Best,
>
> Casey
>
-- 

Jon

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message