metron-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From nickwallen <>
Subject [GitHub] incubator-metron pull request #208: METRON-309 Create a normalcy profiler
Date Wed, 10 Aug 2016 23:48:15 GMT
GitHub user nickwallen opened a pull request:

    METRON-309 Create a normalcy profiler

    ### [METRON-309](
    Created a normalcy profiler, otherwise known as the Metron Profiler.   
    The Profiler is a feature extraction mechanism that can generate a profile describing
the behavior of an entity on a network. An entity might be a server, user, subnet or application.
Once a profile has been generated defining what normal behavior looks-like, it can be used
to build models that identify anomalous behavior.
    This is achieved by summarizing the streaming telemetry data consumed by Metron over sliding
windows. A summary statistic is applied to the data received within a given window. Collecting
this summary across many windows results in a time series that is useful for analysis.
    ### Changes
    This PR contains the following high-level changes to Metron.
    * A Storm topology that generates Profiles. `metron-analytics/metron-profiler`
    * Changes to Ansible to deploy the Profiler topology. `metron-deployment` 
    * A new bolt and associated tooling to perform HBase writes from a Storm topology. `metron-platform/metron-hbase`
    * Additions to the Stellar language to provide arithmetic and control flow logic required
to generate Profiles. `metron-platform/metron-common`
    ### Testing
    Follow the instructions contained in the [README](
to get your very first Profile running.
    ### Documentation
    * Documentation for the Profiler is contained within the [README](
    * The code itself contains plenty of comments and Javadocs.  
    * The design document is attached to the [JIRA](

You can merge this pull request into a Git repository by running:

    $ git pull METRON-309

Alternatively you can review and apply these changes as the patch at:

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #208
commit a77fe3e91601dcf1698271aad1b00e687038b1ed
Author: Nick Allen <>
Date:   2016-07-29T20:53:49Z

    METRON-309 Create a normalcy profiler


If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at or file a JIRA ticket
with INFRA.

View raw message