metron-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Nick Allen <n...@nickallen.org>
Subject Re: [DISCUSS] Ambari Integration
Date Fri, 16 Sep 2016 00:34:26 GMT
Jon - Installing Metron on an isolated network becomes much easier with
Ambari.  You would just mirror the required RPM repositories.  You can then
point Ambari to where your repo lives via the installation wizard.  I've
done quite a few installs via Ambari on an isolated network and it worked
quite well.






On Thu, Sep 15, 2016 at 6:50 PM, Zeolla@GMail.com <zeolla@gmail.com> wrote:

> First of all - very much looking forward to this approach.  I'm not very
> familiar with management packs, but I did read some of the documentation in
> the link you sent.
>
> Not sure if this is already included in a "minimum viable product," but at
> some point I think there needs to be a method of specifying proxies and/or
> internal package repos.  I recently did a Metron 0.2.0 install behind a
> proxy (hence METRON-409 <https://issues.apache.org/jira/browse/METRON-409>)
> and it look me a semi-lengthy amount of time to (1) find all of the
> destinations I needed to request openings for in the proxy, and (2) modify
> the ambari scripts to appropriately use my proxies in the correct way.
>
> I also have a bit of a concern with upgrades and customizations in general
> (Not just how it would work with mpacks).  I have not done any of this to
> date, but I have rebuilt and redeployed a couple of times and I needed to
> modify some of the metron code itself before build/deploy (because of my
> concern with it getting overwritten on upgrade if I just did it directly on
> the cluster).  I would like to see a method of putting in install-specific
> files that modify or overwrite parts of the core metron stack, like changes
> to znodes, parsers, etc.
>
> Regarding not managing sensors with Ambari, I agree.  I run a large bro
> cluster and it is maintained via Puppet and various other mechanisms - no
> need for Ambari to bleed over in my case.
>
> Thanks for the great work.
>
> Jon
>
> On Thu, Sep 15, 2016 at 5:10 PM Casey Stella <cestella@gmail.com> wrote:
>
>> Hi Everyone,
>>
>> I wanted to solicit some discussion around a feature that is fast
>> approaching.  A major pain point in using Metron is installation.  Thus far
>> our only approach to installation has been driven by the developer's needs
>> to construct a virtual environment to test out changes, which lead us to
>> either an ansible installation or a manual installation.
>>
>> Because we want to make sure that the installation of Metron is as easy
>> as possible, we have had some great contributions of an additional
>> approach, installation via Apache Ambari directly.  Our ansible scripts
>> currently rely on Ambari blueprints to set up Hadoop on the cluster that it
>> is deploying on, so it is not a new dependency, but we're working toward a
>> full Ambari management pack
>> <https://cwiki.apache.org/confluence/display/AMBARI/Management+Packs>
>> that will lay down the relevant topologies (parser, enrichment, indexing),
>> configs, bits and their infrastructural dependencies (ES and mysql) and
>> allow the topologies to be started and stopped as minimum viable product.
>>
>> The beginnings of this have started with:
>>
>>    - Ambari Service Definitions for the Parser topologies
>>    <https://github.com/apache/incubator-metron/pull/218>
>>    - Ambari Service Definition for the Indexing Topology
>>    <https://github.com/apache/incubator-metron/pull/222>
>>    - Ambari Service Definition for Elasticsearch
>>    <https://github.com/apache/incubator-metron/pull/223>
>>
>> There will be more to come in the near-term to realize that vision, but
>> we wanted to get some reactions.  Past minimum viable product, what do you
>> guys think we should have and how should it look?
>>
>> Currently we are treating the domain of the ambari installation as from
>> kafka to the indexes, which leaves the sensors unmanaged via ambari.  Is
>> that a good decision?
>>
>> Are there other pain points that you have had around installation that
>> you'd like to see addressed?
>>
>> The purpose of this discussion thread is to let you guys know that we
>> will soon have a new way to install metron, but also to understand what the
>> future requirements are so we, as a community, can address them.
>>
>> Best,
>>
>> Casey
>>
> --
>
> Jon
>



-- 
Nick Allen <nick@nickallen.org>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message