metron-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From nickwallen <...@git.apache.org>
Subject [GitHub] metron pull request #869: METRON-1362 Improve Metron Deployment README
Date Tue, 02 Jan 2018 16:30:00 GMT
Github user nickwallen commented on a diff in the pull request:

    https://github.com/apache/metron/pull/869#discussion_r159261149
  
    --- Diff: metron-deployment/README.md ---
    @@ -1,175 +1,129 @@
    -# Overview
    -This set of playbooks can be used to deploy an Ambari-managed Hadoop cluster containing
Metron services using Ansible. These playbooks target RHEL/CentOS 6.x operating
    -systems.
    -
    -Installation consists of -
    -- Building Metron tarballs, RPMs and the Ambari MPack
    -- Deploying Ambari
    -- Leveraging Ambari to install:
    -  * The required Hadoop Components
    -  * Core Metron (Parsing, Enrichment, Indexing)
    -  * Elasticsearch
    -  * Kibana
    -- Starting All Services
    -
    -## Prerequisites
    -The following tools are required to run these scripts:
    -
    -- [Maven](https://maven.apache.org/)
    -- [Git](https://git-scm.com/)
    -- [Ansible](http://www.ansible.com/) (2.0.0.2 or 2.2.2.0)
    -- [Docker](https://www.docker.com/) (Docker for Mac on OSX)
    -
    -These scripts depend on two files for configuration:
    -
    -- hosts - declares which Ansible roles will be run on which hosts
    -- group_vars/all - various configuration settings needed to install Metron
    -
    -For production use, it is recommended that Metron be installed on an existing cluster
managed by Ambari as described in the Installing Management Pack section below.
    -## Ambari
    -The Ambari playbook will install a Hadoop cluster including the Metron Services (Parsing,
Enrichment, Indexing). Ambari will also install Elasticsearch and Kibana.
    -
    -Currently, the playbooks supports building a local development cluster running on one
node or deploying to a 10 node cluster on AWS EC2.
    -
    -## Vagrant
    -There is a development environment based on Vagrant that is referred to as "Full Dev".
 This installs the entire Ambari/Metron stack. This is useful in testing out changes to the
installation procedure.
    -
    -### Prerequsities
    -- Install [Vagrant](https://www.vagrantup.com/) (5.0.16+)
    -- Install the Hostmanager plugin for vagrant - Run `vagrant plugin install vagrant-hostmanager`
on the machine where Vagrant is
    -installed
    -
    -### Full-Dev
    -Navigate to `metron/metron-deployment/vagrant/full-dev-platform` and run `vagrant up`.
    -
    -## Ambari Management Pack
    -An Ambari Management Pack can be built in order to make the Metron service available
on top of an existing stack, rather than needing a direct stack update.
    -
    -This will set up
    -- Metron Parsers
    -- Enrichment
    -- Indexing
    -- GeoIP data
    -- Optional Elasticsearch
    -- Optional Kibana
    -
    -### Prerequisites
    +This project contains a variety of tools for deploying Apache Metron.  Please refer to
the following sections for more information on the best way to deploy Metron in your environment.
    +
    + * [How do I deploy Metron on a single VM?](#how-do-i-deploy-metron-on-a-single-vm)
    + * [How do I deploy Metron on a large cluster with Ambari?](#how-do-i-deploy-metron-on-a-large-cluster-with-ambari)
    + * [How do I build RPM packages?](#how-do-i-build-rpm-packages)
    + * [How do I build DEB packages?](#how-do-i-build-deb-packages)
    + * [How do I deploy Metron within AWS?](#how-do-i-deploy-metron-within-aws)
    +
    +## How do I deploy Metron on a single VM?
    +
    +#### Who is this for?
    +
    +If you are new to Metron and just want to explore the functionality that it offers, this
is good place to start.  If you are a developer building on Metron, then this is also a great
way to test out your changes.
    +
    +#### What does this do?
    +
    +This option will deploy Metron on a virtual machine running on your computer.  This is
often referred to as the "Full Dev" environment.  
    +
    +#### How do I do it?
    +Follow these [instructions to deploy Metron on a single VM](vagrant/full-dev-platform/README.md).
    +
    +#### Where can I get more information?
    +* [Full Dev Platform](vagrant/full-dev-platform)
    +
    +## How do I deploy Metron on a large cluster with Ambari?
    +
    +#### Who is this for?
    +
    +If you want to see how Metron can really scale by deploying it on your own hardware,
or even in your own cloud-based environment, this is the best option for you.
    +
    +#### What does this do?
    +
    +This creates a Management Pack (MPack) extension for [Apache Ambari](https://ambari.apache.org/)
that simplifies the provisioning, managing and monitoring of Metron on large clusters.  This
allows you to easily install Metron on a large cluster using a simple, guided process.  This
also allows you to monitor cluster health and even secure your cluster with kerberos.
    +
    +#### How do I do it?
    +
    +##### Prerequisites
    +
     - A cluster managed by Ambari 2.4.2+
    -- Metron RPMs available on the cluster in the /localrepo directory.  See [RPMs](#rpms)
for further information.
    -- [Node.js](https://nodejs.org/en/download/package-manager/) repository installed on
the Management UI host
    +- Installable Metron packages (either RPMs or DEBs) available on the cluster, by default,
in the `/localrepo` directory.  
    +- A [Node.js](https://nodejs.org/en/download/package-manager/) repository installed on
the host running the Management and Alarm UI.
    +
    +##### How do I build the Mpack?
    +
    +```
    +cd metron-deployment
    +mvn clean package -Pmpack
    +```
     
    -### Building Management Pack
    -From `metron-deployment` run
    +This results in the Mpack being produced at the following location.
     ```
    -mvn clean package
    +metron-deployment/packaging/ambari/metron-mpack/target/metron_mpack-0.4.2.0.tar.gz
     ```
     
    -A tar.gz that can be used with Ambari can be found at `metron-deployment/packaging/ambari/metron-mpack/target/`
    +##### How do I install the MPack?
     
    -### Installing Management Pack
    -Before installing the mpack, update Storm's topology.classpath in Ambari to include '/etc/hbase/conf:/etc/hadoop/conf'.
Restart Storm service.
    +Copy the tarball to the host where Ambari is installed and run the following command.
     
    -Place the mpack's tar.gz onto the node running Ambari Server. From the command line on
this node, run
     ```
    -ambari-server install-mpack --mpack=<mpack_location> --verbose
    +ambari-server install-mpack --mpack=metron_mpack-0.4.2.0.tar.gz --verbose
     ```
     
    -This will make the services available in Ambari in the same manner as any services in
a stack, e.g. through Add Services or during cluster install.
    -The Indexing / Parsers/ Enrichment masters should be colocated with a Kafka Broker (to
create topics) and HBase client (to create the enrichment and theatintel tables).
    -This colocation is currently not enforced by Ambari, and should be managed by either
a Service or Stack advisor as an enhancement.
    -
    -Several configuration parameters will need to be filled in, and should be pretty self
explanatory (primarily a couple of Elasticsearch configs, and the Storm REST URL).  Examples
are provided in the descriptions on Ambari.
    -Notably, the URL for the GeoIP database that is preloaded (and is prefilled by default)
can be set to use a `file:///` location
    -
    -After installation, a custom action is available in Ambari (where stop / start services
are) to install Elasticsearch templates.  Similar to this, a custom Kibana action to Load
Template is available.
    -
    -Another custom action is available in Ambari to import Zeppelin dashboards. See the [metron-indexing
documentation](../metron-platform/metron-indexing)
    -
    -#### Offline installation
    -Currently there is only one point that would reach out to the internet during an install.
 This is the URL for the GeoIP database information.
    -
    -The RPMs DO NOT reach out to the internet (because there is currently no hosting for
them).  They look on the local filesystem in `/localrepo`.
    -
    -### Current Limitations
    -There are a set of limitations that should be addressed based to improve the current
state of the mpacks.
    -
    -- There is currently no hosting for RPMs remotely.  They will have to be built locally.
    -- Colocation of appropriate services should be enforced by Ambari.  See [#Installing
Management Pack] for more details.
    -- Storm's topology.classpath is not updated with the Metron service install and needs
to be updated separately.
    -- Several configuration parameters used when installing the Metron service could (and
should) be grabbed from Ambari.  Install will require them to be manually entered.
    -- Need to handle upgrading Metron
    -
    -## RPMs
    -RPMs can be built to install the components in metron-platform. These RPMs are built
in a Docker container and placed into `target`.
    -
    -Components in the RPMs:
    -- metron-common
    -- metron-data-management
    -- metron-elasticsearch
    -- metron-enrichment
    -- metron-parsers
    -- metron-pcap
    -- metron-solr
    -- stellar-common
    -
    -### Prerequisites
    -- Docker.  The image detailed in: `metron-deployment/packaging/docker/rpm-docker/README.md`
will automatically be built (or rebuilt if necessary).
    -- Artifacts for metron-platform have been produced.  E.g. `mvn clean package -DskipTests`
in `metron-platform`
    -
    -The artifacts are required because there is a dependency on modules not expressed via
Maven (we grab the resulting assemblies, but don't need the jars).  These are
    -- metron-common
    -- metron-data-management
    -- metron-elasticsearch
    -- metron-enrichment
    -- metron-indexing
    -- metron-parsers
    -- metron-pcap-backend
    -- metron-solr
    -- metron-profiler
    -- metron-config
    -
    -### Building RPMs
    +This will make the Metron available as an installable service within Ambari.
    +
    +#### Where can I get more information?
    +
    +The MPack can allow Metron to be installed and then kerberized, or installed on top of
an already kerberized cluster.  This is done through Ambari's standard kerberization setup.
 Using the MPack is preferred, but instructions for Kerberizing manually can be found at [Kerberos-manual-setup.md](Kerberos-manual-setup.md).
These instructions are reference by the Ambari Kerberos install instructions and include commands
for setting up a KDC.
    +
    +* [Metron Mpack](packaging/ambari/metron-mpack)
    +
    +## How do I build RPM packages?
    +
    +#### Who is this for?
    +If you want to manually install Metron on an RPM-based system like CentOS or if you want
a guided installation process using the Ambari Mpack on an RPM-based system, then this is
the right option for you.
    +
    +#### What does this do?
    +This builds installable RPM packages that allow you to install Metron on an RPM-based
operating system like CentOS.
    +
    +#### How do I do it?
     ```
     cd metron-deployment
     mvn clean package -Pbuild-rpms
     ```
     
    -The output RPM files will land in `target/RPMS/noarch`.  They can be installed with the
standard
    -```
    -rpm -i <package>
    -```
    +The RPM packages will land in `metron-deployment/packaging/docker/rpm-docker/RPMS/noarch`.
  
    +
    +#### Where can I get more information?
    +
    +* [RPM Docker](packaging/docker/rpm-docker)
     
    -## Kibana Dashboards
    +## How do I build DEB packages?
     
    -The dashboards installed by the Kibana custom action are managed by the dashboard.p file.
 This file is created by exporting existing dashboards from a running Kibana instance.
    +**WARNING**: The DEB packages are a recent addition to Metron.  These packages have not
undergone the same level of testing as the RPM packages.  Improvements and more rigerous testing
of these packages is underway and will improve in future releases.  Until then, use these
at your own risk.
     
    -To create a new version of the file, make any necessary changes to Kibana (e.g. on full-dev),
and export with the appropriate script.
    +#### Who is this for?
    +If you want to manually install Metron on a APT-based system like Ubuntu or if you want
a guided installation process using the Ambari Mpack on an APT-based system, then this is
the right option for you.
     
    +#### What does this do?
    +This builds installable DEB packages that allow you to install Metron on an APT-based
operating system like Ubuntu.
    +
    +#### How do I do it?
     ```
    -python packaging/ambari/metron-mpack/src/main/resources/common-services/KIBANA/4.5.1/package/scripts/dashboard/dashboardindex.py
\
    -$ES_HOST 9200 \
    -packaging/ambari/metron-mpack/src/main/resources/common-services/KIBANA/4.5.1/package/scripts/dashboard/dashboard.p
-s
    +cd metron-deployment
    +mvn clean package -Pbuild-debs
     ```
     
    -Build the Ambari Mpack to get the dashboard updated appropriately.
    +The DEB packages will land in `metron-deployment/packaging/docker/deb-docker/target`.
 
    +
    +#### Where can I get more information?
    +* [DEB Docker](packaging/docker/deb-docker)
    +
     
    -Once the MPack is installed, run the Kibana service's action "Load Template" to install
dashboards.  This will completely overwrite the .kibana in Elasticsearch, so use with caution.
    +## How do I deploy Metron within AWS?
     
    -## Kerberos
    -The MPack can allow Metron to be installed and then Kerberized, or installed on top of
an already Kerberized cluster.  This is done through Ambari's standard Kerberization setup.
    +**WARNING**: This is only intended for short-lived testing and development.  This deployment
method has the following severe limitations.
    +* The cluster is not secured in any way. It is up to you to manually secure it.  
    +* The cluster will not survive a reboot.
     
    -### Caveats
    --- End diff --
    
    I will create a README in the metron-mpack/ directory and move all those details there.


---

Mime
View raw message