metron-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (METRON-1717) Relocate Storm Profiler Code
Date Mon, 10 Sep 2018 17:57:00 GMT

    [ https://issues.apache.org/jira/browse/METRON-1717?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16609602#comment-16609602
] 

ASF GitHub Bot commented on METRON-1717:
----------------------------------------

Github user merrimanr commented on a diff in the pull request:

    https://github.com/apache/metron/pull/1187#discussion_r216417073
  
    --- Diff: metron-analytics/metron-profiler-common/README.md ---
    @@ -0,0 +1,386 @@
    +<!--
    +Licensed to the Apache Software Foundation (ASF) under one
    +or more contributor license agreements.  See the NOTICE file
    +distributed with this work for additional information
    +regarding copyright ownership.  The ASF licenses this file
    +to you under the Apache License, Version 2.0 (the
    +"License"); you may not use this file except in compliance
    +with the License.  You may obtain a copy of the License at
    +
    +    http://www.apache.org/licenses/LICENSE-2.0
    +
    +Unless required by applicable law or agreed to in writing, software
    +distributed under the License is distributed on an "AS IS" BASIS,
    +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    +See the License for the specific language governing permissions and
    +limitations under the License.
    +-->
    +# Metron Profiler
    +
    +* [Introduction](#introduction)
    +* [Getting Started](#getting-started)
    +* [Profiles](#profiles)
    +* [Examples](#examples)
    +
    +## Introduction
    +
    +The Profiler is a feature extraction mechanism that can generate a profile describing
the behavior of an entity.  An entity might be a server, user, subnet or application. Once
a profile has been generated defining what normal behavior looks-like, models can be built
that identify anomalous behavior.
    +
    +This is achieved by summarizing the telemetry data consumed by Metron over tumbling windows.
A summary statistic is applied to the data received within a given window.  Collecting these
values across many windows result in a time series that is useful for analysis.
    +
    +Any field contained within a message can be used to generate a profile.  A profile can
even be produced by combining fields that originate in different data sources.  A user has
considerable power to transform the data used in a profile by leveraging the Stellar language.

    +
    +There are three separate ports of the Profiler that share this common code base.
    +* The [Storm Profiler](../metron-profiler-storm/README.md) builds low-latency profiles
over streaming data sets.
    +* The [Spark Profiler](../metron-profiler-spark/README.md) backfills profiles using archived
telemetry.
    +* The [REPL Profiler](../metron-profiler-repl/README.md) allows profiles to be tested
and debugged within the Stellar REPL.
    +
    +## Getting Started
    +
    +1. [Create a profile](../metron-profiler-repl/README.md#getting-started) using the Stellar
REPL. Validate your profile using mock data, then apply real, live data.
    +
    +1. [Backfill your profile](../metron-profiler-spark/README.md#getting-started) using
archived telemetry to see how your profile behaves over time.
    +
    +1. [Deploy your profile](../metron-profiler-storm/README.md#getting-started) to Storm
to maintain a low-latency profile over a streaming data set.
    +
    +1. [Retrieve your profile data](../metron-profiler-client/README.md) using the Stellar
API so that you can build enrichments, alert on abnormalities
    +
    +1. Explore more ways to create [profiles](#more-examples).
    +
    +## Profiles
    +
    +Let's start with a simple example. The following profile maintains a count of the number
of telemetry messages for each IP source address.  A counter is initialized to 0, then incremented
each time a message is received for a give IP source address.  At regular intervals the count
is flushed and stored. Over time this results in a time series describing the amount of telemetry
received for each IP source address.
    --- End diff --
    
    give > given


> Relocate Storm Profiler Code
> ----------------------------
>
>                 Key: METRON-1717
>                 URL: https://issues.apache.org/jira/browse/METRON-1717
>             Project: Metron
>          Issue Type: Sub-task
>            Reporter: Nick Allen
>            Assignee: Nick Allen
>            Priority: Major
>
> The Storm port of the Profiler currently lives in metron-analytics/metron-profiler.  This
should be moved to metron-analytics/metron-profiler-storm.  This would mirror the project
names for the Spark port (metron-profiler-spark) and the REPL port (metron-profiler-repl).
> The package name for the Storm port of the Profiler should be changed to org.apache.metron.profiler.storm.
 This would mimic the package name used for Spark; org.apache.metron.profiler.spark.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message